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This  dissertation  presents  a physical  reflectance  model-based  approach  to  highlight 
separation,  description,  and  inspection  of  3-D  specular  objects.  For  the  highlight 
separation  problem,  we  developed  the  normal  vector  equalization  method.  This  method 
separates  probable  highlights  from  the  raw  images  obtained  from  three  different  locations 
of  light  sources  and  provides  a set  of  highlight-free  (Lambertian)  images  and  a set  of 
highlight-only  (specular)  images. 

For  the  3-D  specular  object  description  problem,  we  used  the  results  of  the 
highlight  separation  to  derive  surface  orientations  and  3-D  depth  map  of  the  object.  The 
surface  orientations  are  calculated  from  the  Lambertian  images  by  solving  the  reflectance 
model  for  Lambertian  surface.  The  3-D  depth  map  is  derived  from  the  surface  orientations 
by  applying  a depth  recovery  method  based  on  a line-integration  technique. 
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The  reflectance  models  employed  in  this  dissertation  include  the  empirical  cosine 
model  for  Lambertian  reflection  and  the  simplified  Torrance-Sparrow  model  for  specular 
reflection.  In  order  to  completely  characterize  the  reflection  models,  it  is  necessary  to  find 
the  reflectance  parameters  representing  the  optical  characteristics  of  the  given  specular 
object.  To  this  end,  a new  method  for  measuring  the  reflectance  parameters  has  been 
developed.  Using  a sample  of  well-defined  shapes  such  as  spheres  or  cylinders  made  from 
the  same  material  as  the  object  under  application,  the  method  extracts  (1)  R which 
represents  a reflectance  constant  of  Lambertian  reflection,  (2)  fC  which  represents  peak 
intensity  of  specular  reflection,  (3)  m which  represents  surface  roughness,  and  (4) 
optionally  y which  represents  the  relative  weighting  factor  between  the  two  reflection 
modes. 

For  the  specular  object  inspection  problem,  we  have  applied  the  frameworks 
discussed  above  to  the  inspection  of  components  assembled  on  a PC  (Printed  Circuit) 
board.  The  inspection  system  has  been  implemented  in  a PC  (Personal  Computer)-based 
image  processing  system  and  applied  to  real  PC  boards  manufactured  in  most  electronic 
industries. 

In  the  Appendix,  we  discuss  an  automated  solder  joint  inspection  system  which 
has  been  developed  as  an  integral  part  of  the  PCB/AID  System  (Automated  Inspection 
and  Diagnosis  System  for  Printed  Circuit  Board  Manufacturing). 
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CHAPTER  1 
INTRODUCTION 

1.1  Statement  of  the  Problem 

The  interpretation  of  3-D  objects  based  on  the  brightness  pattern  of  one  or  more 
images  depends  intricately  on  prior  knowledge  of  surface  properties,  imaging  geometry, 
and  lighting  conditions.  Most  approaches  to  this  problem  have  assumed  that  surfaces  are 
Lambertian,  that  is,  incident  light  is  scattered  by  the  surface  so  that  the  perceived 
brightness  is  independent  of  the  direction  of  view  [Hor75,  Hor79,  Woo78].  However, 
many  practical  tasks  in  industrial  vision  problems  require  interpretation  of  images  of 
specular,  or  shiny,  surfaces  where  the  perceived  brightness  pattern  becomes  a very  strong 
function  of  viewing  direction.  Inspection  and  recognition  of  specular  industrial  parts, 
inspection  of  the  shape  of  solder  joint  surfaces,  and  inspecting  surface  smoothness  of 
metallic  or  plastic  sheets  are  examples  of  the  industrial  tasks  where  surface  specularity 
is  a primary  consideration  [Tou88,  Chi88,  Bar88,  Cap88]. 

In  essence,  the  appearance  of  objects  in  the  real  world  is  related  to  the  complex 
ways  they  reflect  the  light  impinging  on  them  from  the  light  sources.  Different  materials 
reflect  light  in  various  ways,  but  one  thing  common  to  most  materials  is  the  fact  that  they 
are  neither  totally  Lambertian  nor  totally  specular;  it  is  a combination  of  these  two  modes 
of  reflection  that  determines  the  appearance  of  most  objects.  In  short,  it  is  the  sum  of 
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Lambertian  and  specular  reflections  that  determines  the  total  reflection  of  light  off 
surfaces  [Tor67,  C008I,  Ger87]. 

Specular  reflection  (highlight)  has  two  opposing  properties.  Since  highlights 
strictly  constrain  imaging  parameters,  such  as  direction  of  light  source,  surface  normal, 
and  viewing  direction,  highlights  in  image  provide  important  information  which  can  be 
used  to  determine  shape  and  size  of  objects  [Thr83].  For  instance,  if  a line-shaped 
highlight  is  observed  in  an  image,  the  underlying  object  is  strongly  assumed  to  be  a 
cylinder  shape.  In  the  case  where  a highlight  spot  is  observed,  the  object  can  be  assumed 
to  be  spherical  shape  around  the  highlight.  However,  highlights  also  cause  numerous 
problems  for  algorithms  which  require  correspondence  between  images,  such  as  stereo  or 
motion  algorithms.  Observed  highlights  in  one  image  may  disappear  in  another  image 
(stereo)  or  other  images  (motion)  because  of  their  great  dependence  on  imaging  geometry 
[Ger87,  Dre82]  (see  Fig.  1.1).  The  edge  detection  process  employed  in  most  machine 
vision  systems  is  another  example  of  the  troublesome  aspect  of  highlights.  If  we  apply 
an  edge  detector  to  an  image  containing  highlights,  the  edge  detector  will  respond  not 
only  to  real  edges  but  also  to  the  highlights  (see  Fig.  1.2).  This  misleading  edge  informa- 
tion can  cause  confusion  in  the  system  operation.  In  short,  these  two  aspects  of  highlights 
may  be  described  as  the  information  aspect  and  distortion  aspect,  respectively  [Par90b]. 

Most  of  conventional  machine  vision  systems  utilizing  intensity  of  images  use  only 
one  type  of  information  by  assuming  just  one  of  two  reflection  components  in  their 
problem  domains,  namely  Lambertian  reflection  [Hor75,  Woo78]  or  specular  reflection 
[San88,  Hea88a,  Ike81a].  However,  generally,  the  images  of  most  objects  in  the  real  world 
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Fig.  1.1.  Distortion  aspect  of  highlights  in  stereo  vision;  because  of  great  dependence  of 
highlights  on  imaging  geometry,  highlights  observed  in  the  right  image  are  quite  different 
from  those  in  the  left  image. 
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Fig.  1.2.  Distortion  aspect  of  highlights  in  edge  detection;  the  3X3  Prewitt  edge  detection 
operator  responds  not  only  to  real  edges  but  also  to  highlights,  resulting  in  misleading 
edge  information. 
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contain  the  two  modes  of  reflection  as  discussed  above  and  the  information  concealed  in 
both  components  of  reflection  can  be  utilized  as  much  as  possible  to  construct  more 
reliable  and  more  effective  machine  vision  systems  [Par90c,  Kli87,  Kli88,  Nay89]. 

The  rationale  for  this  research  originates  in  the  following  existing  findings: 

(1)  For  most  industrial  objects,  highlights  in  images  are  an  unavoidable  and  important 
phenomenon.  Nevertheless,  they  have  received  little  attention  from  researchers  in 
machine  vision  and  image  processing. 

(2)  Many  of  the  algorithms  which  utilize  the  shape-from-shading  method  often  assume 
a perfect  Lambertian  surface  as  the  surface  of  applicable  objects  in  order  to  avoid 
troublesome  specular  phenomenon. 

(3)  Many  of  conventional  machine  vision  systems  use  a single  channel  of  information 
flow  from  input  image  to  recognition.  Their  overall  system  performances  are  quite 
sensitive  to  any  components  in  the  flow.  In  case  any  failures  occur  in  lower 
processes,  it  is  very  difficult  to  recover  such  failures  in  higher  processes.  The  dual 
channel  processing  scheme  employed  in  this  research  will  greatly  alleviate  such 
problems  since  two  types  of  information  are  available  to  maximize  system 
performance. 

(4)  The  specular  component  as  well  as  the  Lambertian  component  in  images  contains 
important  information  which  can  be  used  to  determine  shape  and  size  of  objects. 

(5)  Only  a few  approaches  to  separate  highlights  from  images  have  been  reported  to 
date.  Most  of  them  utilize  color  vision  techniques  and  have  limitations  in  the 
materials  to  which  those  approaches  can  be  applied  [Kli87,  Ger87]. 
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(6)  Because  of  the  wide  popularity  of  monochromatic  machine  vision  algorithms, 
highlight  separation  techniques  which  utilize  monochromatic  image  data  are 
necessary  for  use  with  other  existing  machine  vision  techniques. 

1.2  Objectives 

The  objective  of  this  research  is  to  provide  some  satisfying  solutions  to  the  above 
issues  by  developing  fundamental  frameworks  for  the  fields  of  highlight  separation,  3-D 
specular  object  description  and  inspection.  In  order  to  achieve  the  goal,  we  attacked  the 
problems  by  conducting  the  following  subtasks: 

(1)  Developing  a versatile  highlight  separation  technique  called  normal  vector 
equalization.  The  technique  uses  common  monochromatic  images,  separates 
probable  highlights  from  the  raw  images,  and  generates  3-D  descriptions  for  the 
specular  objects  in  the  form  of  needle  maps  and/or  3-D  depth  maps  [Par90b, 
Par90e]. 

(2)  Developing  a conceptual  intermediate-level  processing  scheme  called  dual  channel 
processing.  In  the  dual  channel  processing  scheme,  the  two  different  aspects  of 
highlight  are  separately  used  to  maximize  the  system  performance.  In  one  channel, 
the  information  aspect  is  actively  exploited  and  in  the  other  channel  the  distortion 
aspect  is  completely  avoided  [Par90c,  Par91]. 

(3)  Developing  an  analytical  method  for  measuring  optical  characteristics  of  a given 
specular  surface.  The  optical  characteristics  are  represented  by  several  numeric 
data  called  reflectance  parameters.  These  parameters  are  supplied  to  the  normal 
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vector  equalization  method  as  a priori  information  and  can  also  be  applied  to 
computer  graphics  applications  to  render  a realistic  image  or  to  industrial 
inspection  applications  to  check  surface  integrity  [Par90d]. 

(4)  Developing  an  automatic  inspection  system  for  electronic  components  assembled 
on  a PC  (Printed  Circuit)  board.  This  system  is  designed  as  a practical  application 
of  the  framework  discussed  above  [Pai90c,  Par91]. 

(5)  Developing  an  automatic  inspection  system  for  solder  joints.  This  system  works 
as  an  integral  part  of  the  PCB/AID  System  (Automated  Inspection  and  Diagnosis 
System  for  Printed  Circuit  Manufacturing)  [Par89,  Par90a]. 

1.3  Approach 

In  this  dissertation,  we  will  address  the  problems  concerning  highlight  separation, 
3-D  specular  object  description,  and  inspection.  This  domain  of  research  is  extremely 
important  for  other  industrial  applications  of  machine  vision  technologies  such  as  robotics 
and  manufacturing  automation,  since  most  industrial  parts  show  strong  highlights,  as  do 
many  other  man-made  objects. 

As  shown  in  Fig.  1.3,  the  design  of  a machine  vision  system  generally  involves 
several  major  problem  areas.  They  are  (1)  sensing  and  preprocessing  problem  known  as 
low-level  processing;  (2)  feature  extraction  and  scene  analysis  problem  known  as 
intermediate-level  processing;  and  (3)  application  problem  known  as  high-level 
processing.  Throughout  this  dissertation,  we  have  developed  several  unique  methodologies 
for  the  problem  areas.  They  are  (1)  a newly  developed  highlight  separation  technique 
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Fig.  1.3.  Schematic  diagram  of  the  proposed  machine  vision  system  for  specular  object 
inspection  and  recognition. 
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Fig.  1.4.  Removing  the  distortion  aspect  of  highlights  via  highlight  separation;  the  same 
edge  detection  operator  as  in  Fig.  1.2  is  applied  to  the  highlight-free  image  which  is 
obtained  from  the  image  shown  in  Fig.  1.2  using  the  developed  highlight  separation 
technique.  Compared  to  Fig.  1.2,  much  more  reliable  edge  information  can  be  extracted. 
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based  on  the  normal  vector  equalization  method  for  the  preprocessing  problem  area 
[Par90b],  (2)  the  dual  channel  processing  scheme  for  the  feature  extraction  and  scene 
description  problem  area  [Par90c],  and  (3)  a PC  board  inspection  system  for  the 
application  problem  area  [Par91]. 

Using  the  normal  vector  equalization  method,  we  first  separate  the  highlights  from 
the  acquired  raw  images  and  obtain  two  sets  of  highlight-only  (specular)  images  and 
highlight-free  (Lambertian)  images.  The  normal  vector  equalization  method  is  based  on 
the  fact  that  surface  normal  vectors  given  by  the  Lambertian  reflection  model  are  equal 
to  those  given  by  the  simplified  Torrance-Sparrow  model,  which  is  used  to  solve  the 
problem  of  local  shape  from  specularity  [Hea88a].  The  distortion  aspect  of  highlights  is 
completely  removed  in  the  Lambertian  images  (see  Fig.  1.4)  and  the  information  aspect 
of  highlights  is  completely  conserved  in  the  specular  images. 

Secondly,  in  the  dual  channel  processing  scheme,  those  two  sets  of  preprocessed 
images  are  separately  applied  to  higher  level  processes  such  as  edge  detection,  feature 
extraction,  and  shape  description.  For  instance,  edge  contour,  3-D  depth  map,  or  3-D 
surface  orientation  represented  by  EGI  (Extended  Gaussian  Image)  [Hor86,  Ike81b]  can 
be  used  to  globally  constrain  the  shape  of  the  object  for  inspection  and/or  recognition. 
Moreover,  the  features  and  descriptions  extracted  from  the  Lambertian  images  provide 
very  stable  information  regarding  the  global  shape  of  the  object,  because  such  local 
intensity  variations  as  highlights  are  no  longer  present  in  these  highlight-free  images. 
Meanwhile,  the  specular  images  can  provide  sensitive  information  regarding  local  shape 
of  the  object  around  the  highlights.  This  local  information  can  be  used  to  additionally 
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constrain  the  global  shape  of  the  object  and  to  consequently  reduce  the  search  space  for 
establishing  recognition  or  flagging  defects  in  the  subsequent  high-level  processes. 

Finally,  high-level  processes  such  as  inspection  and  recognition  are  performed 
using  the  features  and  descriptions  extracted  from  the  previous  processing  steps. 

1.4  Organization 

Chapter  2 presents  a brief  survey  of  the  imaging  and  light  reflection  models  which 
constitute  a basis  for  this  research.  The  orthographic  projection  model,  empirical 
Lambertian  reflection  model,  and  Torrance-Sparrow’s  specular  reflection  model  are 
presented.  Review  of  the  previous  work  is  also  given  in  this  chapter. 

Chapter  3 presents  the  normal  vector  equalization  method  developed  for  separating 
probable  highlights  and  obtaining  surface  orientations  for  3-D  specular  objects.  Based  on 
the  simplified  Torrance-Sparrow  model  for  specular  reflection  and  the  empirical 
Lambertian  reflection  model,  this  method  uses  a set  of  three  monochromatic  images  taken 
from  three  different  illumination  directions,  separates  probable  highlights  from  those 
images,  and  generates  a set  of  three  Lambertian  images  and  a set  of  three  specular 
images.  Surface  orientations  are  obtained  using  the  Lambertian  images  by  solving  an 
imaging  equation  for  a Lambertian  surface.  A 3-D  depth  map  is  derived  from  the  surface 
orientations  by  applying  a depth  recovery  method  based  on  a line  integration  technique. 
Experimental  results  with  real  objects  as  well  as  synthesized  images  are  discussed. 

Chapter  4 presents  a new  method  for  measuring  reflectance  parameters  which 
represent  optical  characteristics  of  a homogeneous  surface.  The  parameters  include  (1)  R* 
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which  represents  a reflectance  constant  of  Lambertian  reflection,  (2)  if  which  represents 
peak  intensity  of  specular  reflection,  (3)  m which  represents  surface  roughness,  and  (4) 
optionally  y which  represents  the  relative  weighting  factor  between  Lambertian  and 
specular  reflection  modes.  These  reflectance  parameters  are  provided  to  the  normal  vector 
equalization  method  in  which  the  parameters  are  used  to  completely  characterize  the  light 
reflection  models.  The  method  uses  a spherical  specimen  which  is  made  from  the  same 
material  as  the  object  under  application,  takes  three  images  with  three  different  directions 
of  light  sources,  and  extracts  the  above  mentioned  parameters  automatically.  Experimental 
results  with  real  objects  as  well  as  synthesized  objects  are  also  discussed. 

Chapter  5 presents  the  design  of  an  automated  inspection  system  for  electronic 
components  on  a PC  board.  The  inspection  system  has  been  developed  as  an  example  of 
the  practical  application  of  the  basic  research  work  presented  in  Chapter  3 and  Chapter 
4.  Therefore,  the  inspection  system  employs  several  unique  methodologies  discussed  in 
the  previous  chapters.  They  are  (1)  the  normal  vector  equalization  method  to  separate 
probable  highlights  and  obtain  two  forms  of  image  information,  i.e.  Lambertian  images 
and  specular  images,  (2)  the  dual  channel  processing  scheme  to  maximize  the  utilization 
of  the  information  concealed  in  those  two  types  of  images,  and  (3)  a decision-making 
method  based  on  a proximity  measure.  The  system  has  been  tested  for  several  real  PC 
boards.  Experimental  results  are  also  discussed. 

Chapter  6 summarizes  the  major  accomplishments  in  this  dissertation  and  provides 
some  suggestions  for  further  research. 
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In  the  Appendix,  we  present  the  PCB/AID  System  (Automated  Inspection  and 
Diagnosis  System  for  PC  Board  Manufacturing)  in  which  the  component  inspection 
system  discussed  in  Chapter  5 plays  an  important  role.  The  roles  of  the  PCB/AID  System 
are  (1)  automating  visual  inspection  processes  for  solder  joints  and  electronic  components 
which  are  usually  performed  by  human  inspectors  in  most  industries  and  (2)  utilizing  the 
inspection  results  provided  from  the  automated  inspection  systems  in  a fault  diagnosis 
system  to  automatically  control  the  manufacturing  processes.  The  inspection  part  of  the 
PCB/AID  System  consists  of  two  subsystems.  They  are  (1)  a solder  joint  inspection 
system,  and  (2)  a component  inspection  system.  Of  these  subsystems,  the  component 
inspection  system  is  discussed  in  Chapter  5 and  the  solder  joint  inspection  system  is 
discussed  in  the  Appendix. 


CHAPTER  2 

PREREQUISITES  AND  REVIEW  OF  PREVIOUS  WORK 


2. 1 Image  Formation  Model 

In  order  to  utilize  intensity  images  to  derive  the  shape  of  an  object  or  to  separate 
probable  highlights  from  the  intensity  images,  it  is  essential  to  understand  a model  of 
image  formation  which  describes  how  the  image  intensity  at  a pixel  is  determined.  In  Fig. 
2.1,  a basic  model  for  image  formation  is  illustrated.  It  includes  a light  source,  a surface, 
and  an  image  plane.  It  is  customary  to  place  a coordinate  system  so  that  the  viewer’s 
central  line  of  sight  (optical  axis)  is  in  the  z-axis,  and  the  picture  plane  is  in  the  x-y 
plane.  A small  surface  patch  at  (x,y,z)  is  illuminated  by  the  light  source  and  some  part 
of  the  reflected  light  is  recorded  at  a corresponding  pixel  position  in  the  image.  The 
intensity  recorded  at  a pixel  is  generally  dependent  on  the  illumination  position,  surface 
orientation,  viewer  position,  and  reflectance  characteristics  of  the  surface.  However,  for 
simplicity,  we  can  assume  that  the  incident  illumination  is  parallel  and  constant  at  each 
surface  position  and  that  the  image  projection  is  orthographic.  These  assumptions 
correspond  to  the  case  that  the  illumination  source  and  the  viewer  are  both  far  enough 
from  the  object  surface  [Coh82]. 

Generally,  surface  orientations  in  3-D  space  can  be  represented  by  means  of  a 
gradient.  If  a surface  is  defined  explicitly  by  an  equation  of  the  form, 
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V : view  vector 
S:  source  vector 
N:  surface  normal  vector 

ff=(V+S)/|V+S|:  unit  angular  bisector  of  V and  S 
a=cos  \N  H):  angle  between  N and  H 
§=cos\N  V):  emittance  angle 
<}>i=cos'1(^V  iS):  incidence  angle 
4»g=cos  ‘(S  V):  phase  angle 


Fig.  2.1.  An  image  formation  model. 
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z - A*,y)  <21) 

then,  surface  normal  vectors  representing  its  orientation  in  3-D  space  can  be  described 
using  a gradient  of  a scalar  surface  defined  as,  F(x,y,z)Afix,y)-z=0.  That  is, 


a ,dF  dF  3Fa  ,dz  dz  n 
Gradient  of  F = AF  = ,-l) 

ox  dy  oz  ox  dy 


The  unit  surface  normal  vector  N is  given  by, 


N = tfVVQ  = 


AF 

|AF| 


( * * _l) 

^ dx’  dy’  ’ 


(*)2+(*)2+l 

\|  a*  ay 


By  defining  p=dz/dx  and  q=dz/dy,  we  have  from  (2.3), 


(2.2) 


(2.3) 


N. 


(2.4.a) 


\lp2+q2+ 1 


(2.4.b) 


W = - _1  (2.4.c) 

\lp2+q2+ 1 

In  (2.4),  the  surface  normal  of/fx.y)  at  any  point  (jt,y)  can  completely  be  described  using 
p and  q.  The  quantity  (p,q)  is  defined  as  the  gradient  of  fix, y).  Gradient  space  is  the  two 
dimensional  vector  space  spanned  by  the  set  of  all  such  gradients  (p,q).  The  concept  of 
gradient  space  has  been  widely  used  to  solve  many  of  the  machine  vision  problems 
[Coh82,  Mac73,  Kan80]. 
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The  fundamental  fact  upon  which  the  photometric  method  is  based  is  that  isotropic 
surfaces  have  a unique  value  of  intensity  for  a particular  surface  orientation  and 
reflectance  factor.  In  other  words,  given  a reflectance  factor  which  accounts  for  the 
optical  properties  of  surface  microstructure,  measured  intensity  depends  solely  upon 
surface  orientation  and  illumination. 

A surface  photometric  function  Q(<t>„<}>e,<j>g)  for  a single  source  can  be  defined  in 
terms  of  the  angles  <j>j,  <)>e,  and  <))g.  These  angles  are  called  the  incident  angle  (<(>,), 
emittance  angle  (<|>e),  and  phase  angle  (<j>g).  These  angles  quantify  the  relationships 
between  the  source  vector,  view  vector,  and  surface  normal  vector  as  shown  in  Fig.  2.1. 

Given  the  condition  for  a single  source  of  illumination,  a mapping  can  be 
constructed  between  intensity  and  gradient  space  coordinates.  This  mapping,  denoted  by 
R(p,q),  is  called  the  reflectance  map.  From  the  reflectance  map,  Woodham  [Woo78] 
illustrates  how  a single  intensity  value  constrains  the  p and  q values  of  the  surface  normal 
vectors  to  a particular  iso-brightness  contour.  The  goal  of  shape  from  shading  is  to 
identify  which  (p,q)  value  on  the  contour  is  the  correct  surface  gradient  [Hor86]. 

2.2  Light  Reflection  Model 

In  the  imaging  model  shown  in  Fig.  2.1,  light  energy  reflected  by  the  surface  in 
the  direction  of  the  camera  causes  an  image  of  the  surface  to  be  formed  in  the  camera. 
For  a given  orientation  of  the  surface  and  direction  of  the  point  illumination  source,  the 
amount  of  light  energy  reflected  by  the  surface  in  a particular  direction  is  determined  by 
the  reflectance  properties  of  the  surface.  The  reflectance  model  of  a large  number  of 


18 


surfaces  comprises  two  components,  namely,  the  Lambertian  (diffusive)  component  and 
the  specular  (glossy)  component.  In  general,  the  energy  of  light  reflected  by  a surface  in 
any  direction  is  a combination  of  the  Lambertian  and  specular  components. 

In  Fig.  2.2,  a microscopic  mechanism  of  light  reflection  is  presented  [Lee90]. 
When  light  illuminates  a surface,  part  of  it  is  reflected  immediately  from  the  surface,  and 
part  of  it  penetrates  into  the  material  body  and  is  reflected  back  to  air  after  being 
absorbed  within  the  material.  The  former  reflection  component  is  called  specular 
reflection  and  the  latter  is  called  Lambertian  reflection.  A general  reflectance  model 
derived  from  this  observation  can  be  described  as  the  weighted  sum  of  the  two  reflection 
components  [C008I,  Bli77,  Lee90].  That  is, 

I(x,y)  = yl{x,y)+(l-y)lt(x,y)  (2-5) 

where  I(x,y)  represents  the  perceived  image  intensity,  Ijx,y)  represents  the  intensity  of  the 
Lambertian  component,  and  IJx.y)  represents  the  intensity  of  the  specular  component.  The 
coefficient  y denotes  the  relative  weighting  factor  between  those  two  reflection 
components  and  has  a value  of  0<y<l.  For  a more  specular  surface,  y becomes  close  to 
zero  and  for  a more  Lambertian  surface,  y becomes  close  to  one. 

The  Lambertian  component  I/x,y)  is  generally  described  by  a cosine  function  of 
the  angle  between  surface  normal  vector  N and  light  source  vector  S while  it  is 
independent  of  the  viewing  direction  V [Hor86]: 


Ifay)  = Rcos(<b)  = R(S-N) 


(2.6) 


incident  light 


specular  component 


Fig.  2.2.  Microscopic  light  reflection  model  [Lee90]. 
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where  R denotes  surface  reflection  factor  for  a Lambertian  surface.  It  represents  a peak 
intensity  of  a Lambertian  image  at  which  the  incidence  angle  is  equal  to  zero. 

The  specular  component  IJx,y)  is  at  a maximum  when  the  incidence  angle  (J),  is 
equal  to  the  emittance  angle  <)>e  and  the  vectors  N,  V,  and  S are  all  in  the  same  plane. 
This  condition  means,  in  other  words,  that  N becomes  the  bisector  of  V and  S,  i.e.  ot=0. 
For  a perfectly  specular  surface,  reflections  are  detected  only  for  the  locations  where  the 
above  condition  is  satisfied.  In  practice,  however,  due  to  surface  irregularities,  specular 
reflections  are  detected  even  for  source  directions  that  are  in  close  proximity  to  the  case 
that  oc=0.  Therefore,  a more  realistic  model  for  specular  reflection  would  be  one  that 
maximizes  reflected  intensity  when  a=0  and  rapidly  decreases  the  reflected  intensity  as 
a deviates  from  zero. 

An  empirical  model  for  specular  reflection  has  been  developed  by  Phong  [Pho75] 
for  computer  graphics.  This  model  represents  the  specular  component  of  reflection  by 
powers  of  the  cosine  of  the  angle  between  the  perfect  specular  direction  and  the  line  of 
sight.  Thus,  Phong’s  model  is  capable  of  predicting  specularities  which  extend  beyond  a 
single  point.  While  Phong’s  model  gives  a reasonable  approximation  which  is  useful  in 
some  contexts,  the  parameters  of  this  model  have  no  physical  meaning. 

The  Torrance -Sparrow  model  [Tor67]  is  a more  refined  model  of  specular 
reflection.  This  model  assumes  that  a surface  is  composed  of  small,  randomly-oriented, 
mirror-like  facets.  Only  facets  with  their  normal  vectors  in  the  direction  of  H contribute 
to  Is.  The  model  also  quantifies  the  shadowing  and  masking  of  facets  by  adjacent  facets 
using  a geometrical  attenuation  factor.  The  resulting  specular  model  is  [C008I]; 
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Is  = FDA  (2.7) 

where  F represents  the  Fresnel  coefficient,  D represents  the  facet  orientation  distribution 
function,  and  A represents  the  adjacent  geometrical  attenuation  factor. 

The  Fresnel  coefficient  F models  the  amount  of  light  which  is  reflected  from 
individual  facets.  In  general,  F depends  on  the  incidence  angle  and  the  complex  index  of 
refraction  of  the  reflecting  material.  Cook  and  Torrance  [C008I]  have  shown  that  to 
synthesize  realistic  images,  F must  characterize  the  color  of  the  specularity.  The  Fresnel 
equations  predict  that  F is  a nearly  constant  function  of  the  incidence  angle  for  the  class 
of  materials  with  a large  extinction  coefficient  [Hea88].  This  class  of  materials  includes 
all  metals  and  many  other  materials  with  a significant  specular  reflection  component. 

The  distribution  function  D describes  the  orientation  of  the  micro  facets  relative 
to  the  average  surface  normal  N.  Blinn  [Bli77]  and  Cook  and  Torrance  [C008I]  discuss 
various  distribution  functions.  All  of  these  functions  are  very  similar  in  shape.  In 
agreement  with  Torrance  and  Sparrow,  we  use  the  Gaussian  distribution  function  given 
by; 

D = Kexp(-— ) (2.8) 

m2 

where  K is  a normalization  constant.  Thus,  for  a given  a,  D is  proportional  to  the  fraction 
of  facets  oriented  in  the  direction  of  H.  The  constant  m indicates  surface  roughness  and 
is  proportional  to  the  standard  deviation  of  the  Gaussian.  Small  values  of  m describe 
smooth  surfaces  for  which  most  of  the  specular  reflection  is  concentrated  in  a single 
direction.  Large  values  of  m are  used  to  describe  rougher  surfaces  with  larger  differences 
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in  orientation  between  nearby  facets.  These  rough  surfaces  produce  highlights  which 
spread  out  on  the  reflecting  surface. 

The  factor  A given  in  (2.7)  quantifies  the  effect  of  a geometrical  attenuation  factor 
B corrected  for  foreshortening  by  dividing  by  (N»V). 


A = 


B 

N-V 


(2.9) 


where  B is  derived  by  Torrance  and  Sparrow  in  [Tor67].  They  assume  that  each  specular 
facet  makes  up  one  side  of  a symmetric  v-groove  cavity.  From  this  assumption,  they 
examine  the  various  possible  facet  configurations  which  correspond  to  shadowing  or 
masking.  The  expression  is; 


B - (2.10) 

\ VH  VH  j 

The  foreshortening  effect  taken  into  account  by  employing  ( N»V)  for  the 
denominator  of  (2.9)  can  be  explained  as  follows.  Let  <(>e  be  the  angle  between  N and  V. 
As  <J)e  increases  from  0 to  90  degrees,  the  viewer  gradually  sees  a larger  part  of  the  reflec- 
ting surface  in  a unit  area  in  the  view  plane.  Therefore,  as  <(>„  becomes  larger,  there  are 
correspondingly  more  surface  facets  which  contribute  to  the  intensity  perceived  by  the 
viewer.  Torrance  and  Sparrow  took  this  phenomenon  into  account  in  (2.9)  by  dividing  by 
N»V. 


The  full  generality  of  the  original  Torrance-Sparrow  model  has  been  simplified  by 
Healey  and  Binford  with  reasonable  assumptions  and  the  resultant  simplified  Torrance- 
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Sparrow  model  has  been  used  to  solve  the  problem  of  local  shape  from  specularity 
[Hea88a].  The  assumptions  are  as  follows. 

The  first  assumption  is  that  F is  a constant  with  respect  to  viewing  geometry.  This 
is  a very  good  approximation  for  metals  and  for  many  other  materials.  The  second 
assumption  is  that  the  exponential  factor  in  (2.8)  changes  much  faster  than  any  other 
terms  of  A.  Therefore,  except  for  a small  range  of  angles  near  grazing  incidence,  A can 
be  considered  constant  across  the  highlight.  Hence,  the  final  form  of  the  simplified 
Torrance-Sparrow  model  is  described  as  [Hea88a]: 

Is(x,y)  = tfex p[“!M]  (2.11) 

m2 

where  AT  is  a constant  representing  peak  highlight  intensity. 

Combining  (2.5),  (2.6),  and  (2.1 1),  we  have  the  following  equations  for  the  image 
intensity  which  contains  those  two  components  of  reflection,  simultaneously: 

I{x,y)  = y/?5^(x,y)+(l-Y)/:exp[-^^] 

” (2.12) 

= RMS-N(x,y)+K*ex p[J?JM] 

m2 

where  R'=yR  and  K * =(l-y)K.  They  represent  peak  Lambertian  intensity  and  peak  specular 
intensity,  respectively,  for  the  surface  in  which  both  reflection  modes  are  present.  Most 
of  the  real  objects  can  be  categorized  as  shown  in  Fig.  2.3  based  on  the  values  of  the 
parameters,  y and  m in  (2.12).  For  highly  specular  objects,  small  values  of  y and  m are 
used  to  represent  such  surfaces  because  the  amount  of  Lambertian  reflection  is  relatively 
small  and  the  surface  is  quite  smooth  and  slick.  For  medium  specular  objects,  medium 


m (radian) 


Fig.  2.3.  Degree  of  specularity  quantified  from  y and  m. 
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values  of  y and  m are  used.  Similarly,  for  near  Lambertian  objects,  large  values  of  y and 
m are  used.  For  most  real  objects,  such  unusual  cases  of  small  y and  large  m or  large  y 
and  small  m hardly  occur  because  m,  which  represents  the  surface  irregularity,  is  more 
or  less  related  to  y,  which  represents  the  relative  strength  between  Lambertian  and 
specular  reflection  components  [C008I]. 

(2.12)  contains  several  unknowns.  They  can  be  categorized  as  (1)  surface 
reflectance  parameters  - R',  K*,  and  m,  (2)  imaging  geometry  - S,  V,  and  H,  and  (3) 
surface  orientation  of  the  object  - N.  Of  these  unknowns,  imaging  geometry  and 
reflectance  parameters  can  be  obtained  using  the  method  discussed  in  Chapter  4.  If  these 
parameters  are  known  a priori,  the  only  unknown  becomes  N,  for  which  we  can  solve 
(2.12).  In  Chapter  3,  we  will  show  how  to  use  (2.12)  to  solve  for  surface  orientation  N 
and  to  separate  specular  reflection  component  IJx,y)  from  the  original  raw  image  I(x,y). 

2.3  Review  of  Previous  Work 

Since  the  late  of  1970s,  several  approaches  for  obtaining  surface  orientations  of 
the  specular  objects  and  separating  or  identifying  probable  highlights  have  been  developed 
in  connection  with  the  specular  object  understanding  problem.  Here,  we  will  discuss  the 
most  representative  frameworks  for  (1)  obtaining  surface  orientation  of  3-D  specular 
objects  and  (2)  separating  or  identifying  highlights  from  an  image  of  these  objects. 
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2.3.1  Surface  Orientation  of  Specular  Objects 

In  Ikeuchi’s  work  [Ike81a],  the  orientation  of  patches  on  the  surface  of  a specular 
object  is  determined  from  multiple  images  taken  with  different  illumination,  but  from  the 
same  viewing  position.  The  method,  referred  to  as  photometric  stereo,  is  implemented 
using  a look-up  table  which  is  based  on  numerical  inversion  of  reflectance  maps.  The 
author  uses  a distributed  light  source  obtained  by  uneven  illumination  of  a diffusely- 
reflecting  planar  surface  to  illuminate  the  specular  object  and  take  its  images. 

This  approach  has  been  proven  to  work  well  for  almost  perfectly  specular  objects, 
even  though  there  are  some  difficulties  in  applying  it  to  a real  environment.  Moreover, 
since  the  author  assumed  only  perfectly  specular  reflection  from  the  object  under 
application,  its  application  areas  are  limited  to  only  highly  specular  objects. 

Coleman  and  Jain  [Col82]  extended  the  photometric  stereo  method  to  four-source 
photometry  for  obtaining  the  3-dimensional  shape  of  visually  textured  and  specular 
surfaces  from  intensity  images.  Among  the  four  images,  three  images  are  chosen  to  avoid 
the  highlights.  In  other  words,  three  images  which  do  not  show  any  prominent  highlights 
are  selected  from  the  four  images.  With  those  three  images,  surface  normal  vectors  are 
calculated  using  the  model  of  Lambertian  surface.  The  authors  assume  that  the  selected 
three  images  consist  of  pure  Lambertian  components,  even  though  they  may  still  have 
some  specular  components.  This  approach  does  not  eliminate  the  specular  component 
thoroughly,  but  merely  avoids  it.  Under  such  circumstances,  accurate  surface  normal 
vectors  may  not  be  recovered  using  the  simple  Lambertian  reflection  model. 
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Sanderson  et  al.  [San88]  developed  an  approach  called  structured  highlight  for 
illumination  and  imaging  of  specular  surfaces  which  produces  3 -dimensional  shape 
information.  The  structured  highlight  approach  uses  a scanned  array  of  point  sources  and 
images  of  the  resulting  reflected  highlights  to  compute  local  surface  height  and 
orientation.  A prototype  Structured  Highlight  INspection  sYstem,  called  SHINY,  has  been 
implemented  and  demonstrated  the  determination  of  surface  shape  for  several  specular  test 
objects  including  solder  joints.  In  this  work,  the  resolution  of  recovered  surface  normal 
vector  is  directly  proportional  to  the  number  of  point  sources  in  the  scanned  array.  The 
number  of  required  images  is  equal  to  that  of  point  sources.  In  order  to  obtain  a 
practically  useful  resolution  of  recovered  surface  normal  vectors,  a few  dozen  point 
sources  and  images  are  required.  Hence,  huge  amounts  of  image  data  must  be  processed 
to  obtain  surface  orientations  with  this  approach. 

Healey  and  Binford  [Hea88a]  adopted  the  Torrance-Sparrow  model  to  obtain  local 
curvatures  of  a highly  specular  object.  From  this  model,  they  derived  relationships 
between  the  properties  of  a specular  feature  in  an  image  and  local  properties  of  the  cor- 
responding surface.  A shape-from-specularity  system  has  been  implemented  to  test  their 
approach.  The  performance  of  the  system  is  demonstrated  by  careful  experiments  with 
several  specular  objects. 

Nayar  and  Ikeuchi  [Nay89]  developed  a method  for  determining  the  shape  of 
surfaces  whose  reflectance  properties  may  vary  from  pure  Lambertian  to  perfectly 
specular  without  prior  knowledge  of  the  relative  strength  of  the  Lambertian  and  specular 
components  of  reflection.  The  object  surface  is  illuminated  using  extended  light  sources 
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and  the  surface  is  viewed  from  a single  direction.  An  extended  light  source  is  generated 
by  placing  a layer  of  light-diffusing  material  between  the  object  and  a point  source  of 
light.  An  array  of  extended  sources  is  sequentially  scanned,  and  for  each  source  an  image 
of  the  object  is  digitized  and  stored  in  memory.  An  extraction  algorithm  uses  the  set  of 
image  intensity  values  at  each  surface  points,  resulting  from  the  array  of  sources,  to 
compute  orientations  as  well  as  relative  strength  of  the  Lambertian  and  specular 
components  of  reflection. 

2.3.2  Identification  and  Separation  of  Highlights 

Klinker  et  al.  [Kli87]  proposed  an  approach  to  solve  the  problem  of  identifying 
highlights  in  a color  image.  The  basis  of  their  work  is  a simple  mathematical  model  of 
color  reflectance,  called  the  dichromatic  reflection  model.  With  this  model,  an  algorithm 
is  proposed  for  computing  the  intrinsic  images  of  Lambertian  and  specular  components 
over  a set  of  [R,G,B]  values  of  pixels  corresponding  to  a single  surface  in  the  image.  The 
problems  with  this  approach  are  (1)  it  can  only  be  applied  to  dielectric  materials  such  as 
plastic  and  rubber  because  the  dichromatic  reflectance  model  works  only  for  such 
materials  and  (2)  it  requires  that  the  color  of  the  object  must  be  different  from  the  color 
of  the  light  source. 

Gershom  et  al.  [Ger87]  discussed  the  problem  of  identifying  probable  highlights 
in  a color  image.  The  solution  they  have  proposed  is  based  on  the  physics  of  interaction 
between  light  and  materials,  as  formulated  in  models  of  optics  and  reflection.  Based  on 
these  models,  a prediction  of  the  behavior  of  highlights  is  made,  and  is  observed  in  real 
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images.  Based  on  the  observed  phenomenon,  namely,  the  shift  in  the  color  of  objects  from 
the  Lambertian  reflection  to  the  specular  reflection,  they  have  developed  an  algorithm 
designed  to  detect  these  shifts.  The  algorithm  requires  a segmentation  of  the  image  into 
regions,  so  that  each  region  can  be  related  to  its  neighbors  to  verify  whether  they  share 
the  shift  in  color.  Since  the  color  shifts  can  be  characterized  by  a feature  in  color  space, 
the  algorithm  is  designed  to  detect  such  relationships  between  regions,  and  augmented 
with  rules  that  employ  information  about  the  expected  color  of  highlights  in  the  scene. 
The  algorithm  has  been  tested  with  real  images  and  produced  useful  results  in  detecting 
the  location  of  probable  highlights. 

In  contrast  to  the  previous  two  approaches  which  utilize  a color  image,  Wolff 
[Wol89a]  announced  a highlight  separation  technique  which  utilizes  the  polarization 
properties  of  reflected  light.  The  method  is  based  on  the  fact  that  the  polarization  states 
for  two  components  of  reflection,  i.e.  Lambertian  and  specular,  are  different.  By 
separating  out  the  parallel  and  perpendicular  components  of  polarization  for  reflected  light 
using  a polarizing  filter,  a respective  2-D  plot  of  points  is  generated.  Each  point 
coordinate  pair  corresponds  to  the  parallel  and  perpendicular  polarization  magnitude  at 
a pixel,  respectively.  These  plots  of  points  form  linear  clusters,  and  the  measured  value 
of  the  slopes  of  respective  approximating  lines  is  crucial  to  determine  the  magnitude  of 
the  Lambertian  and  specular  components  of  reflection.  It  is  also  seen  that  the  slope  of 
these  linear  clusters  can  be  used  to  classify  a material  as  to  whether  it  is  metal  or 
dielectric  [Wol89b].  This  technique  works  for  both  dielectric  and  metallic  surfaces. 
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2.3.3  Conclusions  from  Previous  Work 

Most  of  highlight  separation  techniques  announced  to  date  utilize  a color  image 
[Kli87,  Ger87].  This  is  because  their  approaches  are  essentially  based  on  a color 
difference  between  Lambertian  and  specular  components  of  reflection.  These  techniques 
can  work  successfully  for  objects  made  from  dielectric  materials  since  these  materials 
reflect  the  impinging  light  in  such  a way  that  those  two  components  of  reflection  make 
different  colors.  However,  most  metallic  surfaces  do  not  show  such  color  difference  and 
therefore  it  becomes  impossible  to  apply  those  highlight  separation  techniques  to  those 
surfaces.  Moreover,  those  techniques  can  only  be  used  for  highlight  separation  purposes, 
whereas  the  approach  presented  in  this  dissertation  can  be  applied  not  only  to  highlight 
separation  but  also  to  surface  normal  vector  generation,  which  leads  to  the  3-D  description 
of  the  specular  object. 

The  highlight  separation  technique  developed  by  Wolff  [Wol89a]  can  overcome 
some  of  the  drawbacks  mentioned  above  by  exploiting  polarization  properties  in 
Lambertian  and  specular  components  of  reflection.  However,  it  can  also  be  applied  only 
for  highlight  separation  purposes.  Moreover,  the  necessity  for  mechanical  change  of 
polarization  filter  mounted  on  a camera  lens  seems  to  make  it  difficult  to  completely 
automate  the  whole  operational  processes. 

Most  of  the  previous  work  [Ike81a,  San88,  Hea88a]  for  obtaining  surface 
orientations  of  3-D  specular  objects  can  only  be  applied  to  a perfectly  specular  object 
since  they  do  not  take  into  account  the  Lambertian  component  of  reflection.  Therefore, 
their  application  areas  are  very  limited. 
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Coleman  and  Jain  [Col82]  and  Nayar  and  Ikeuchi  [Nay89]  have  proposed  methods 
for  obtaining  surface  orientations  of  objects  which  show  Lambertian  reflection  as  well  as 
specular  reflection.  In  Coleman  and  Jain’s  work,  the  specular  reflection  component  is 
avoided  by  discarding  an  image  which  includes  very  high  intensity  from  a total  of  four 
images  to  simplify  the  calculation  of  surface  orientations.  It  can  only  provide  surface 
orientations,  whereas  the  approach  discussed  in  this  dissertation  can  provide  separated 
Lambertian  and  specular  components  of  reflection  as  well. 

The  approach  presented  in  this  dissertation  and  Nayar  and  Ikeuchi ’s  approach  have 
some  similarities  and  these  two  approaches  completely  resolve  some  of  the  limitations 
that  most  of  the  previous  work  has  not  overcome  in  the  following  senses:  (1)  They  can 
be  used  to  obtain  surface  orientations  as  well  as  relative  strength  of  Lambertian  and 
specular  components  of  reflection.  (2)  They  can  be  applied  to  a general  surface  ranging 
from  pure  Lambertian  to  highly  specular.  (3)  Common  monochromatic  images  are  used 
in  the  approaches  and  therefore  the  color  of  light  source  does  not  need  to  be  different 
from  that  of  a surface.  (4)  No  mechanical  change  of  filters  is  necessary  and  all  of 
operational  processes  can  be  controlled  electrically. 

Nayar  and  Ikeuchi’s  work,  however,  has  some  limitations  in  practical  application 
in  the  following  senses:  (1)  It  uses  the  simplest  specular  reflection  model  that  can  only 
account  for  the  highest  intensity  point  in  measured  highlight  which  in  most  cases  has 
some  area.  As  a result  of  this,  a number  of  light  sources  and  images  are  used  to  obtain 
practically  useful  results  especially  for  highly  specular  surfaces.  (2)  Because  of  the 
complexity  of  the  sensing  apparatus,  system  development  and  experiments  are  only 
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performed  for  2-D  cases  in  order  to  demonstrate  its  feasibility.  (3)  Since  a number  of 
images  must  be  processed,  execution  time  of  their  system  is  the  main  problem.  The 
authors  claimed  that  1 msec  per  image  pixel  is  the  execution  speed  of  the  algorithm  on 
a 3/60  SUN  workstation.  In  their  2-D  experimental  system,  they  used  4 light  sources  per 
direction.  If  they  expand  the  system  to  cover  3-D  by  employing  8-directional  light 
sources,  a total  of  4X8=32  light  sources  will  be  required.  For  the  images  of  256X256 
resolution,  overall  processing  time  will  be  10'3X3 2X256X256=2097  sec.=35  min.  As  the 
authors  have  pointed  out,  the  developed  algorithm  must  be  improved  to  accommodate  the 
strict  requirement  of  processing  speed  in  real-world  applications. 


CHAPTER  3 

HIGHLIGHT  SEPARATION  AND  SHAPE  OF  3-D  SPECULAR  OBJECTS 

3.1  Introduction 

Most  research  in  machine  vision  has  heavily  relied  on  the  information  contained 
in  an  image  or  images.  Obviously,  information  about  the  3-D  shape  of  a given  surface 
will  greatly  enhance  the  system  performance.  Several  approaches  are  available  to  obtain 
3-D  depth  information  about  a surface.  They  are  largely  divided  into  the  direct  method 
and  indirect  method  [Col82].  The  direct  method  directly  measures  the  range  (relative 
distance  between  object  and  sensor).  The  time-of-flight  technique  based  on  phase  shift  of 
laser  pulse  is  a typical  example  of  the  direct  method  for  measuring  distance  [Pau90]. 

The  indirect  method  determines  the  distance  by  measuring  parameters  derived 
from  intensity  images  of  a given  object.  Examples  in  this  category  include  (1)  the  stereo 
vision  technique,  which  utilizes  triangulation  between  the  correspondence-points  [Bal82, 
Fu87],  (2)  optical  flow  technique,  which  utilizes  the  flow  of  image  points  from  one  frame 
to  the  next  frame,  and  (3)  shape-from-shading  technique,  which  utilizes  the  relationship 
between  image  intensity  (shading)  and  surface  orientation  (shape)  [Hor75,  Hor79]. 

In  this  chapter,  we  will  address  the  problem  of  obtaining  surface  orientations 
and/or  3-D  depth  map  of  a specular  object  by  separating  highlight  from  raw  images. 
Based  on  the  light  reflection  model  given  in  (2.12)  for  a general  specular  surface,  we  will 
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derive  a computational  method  for  highlight  separation  called  normal  vector  equalization. 
This  method  is  based  on  the  fact  that  the  surface  orientations  given  by  the  Lambertian 
reflection  model  are  equal  to  those  given  by  the  specular  reflection  model  for  the  general 
surfaces  in  which  those  two  reflection  modes  are  present. 

Using  the  normal  vector  equalization  method,  we  will  separate  probable  highlights 
from  the  raw  images  which  are  acquired  from  three  different  locations  of  light  sources. 
Surface  orientations  are  derived  from  the  Lambertian  images,  in  which  the  highlights  are 
completely  removed,  by  solving  the  empirical  Lambertian  reflection  model.  A 3-D  depth 
map  is  obtained  by  applying  the  surface  orientations  to  a depth  recovery  method  based 
on  a line-integration  technique.  The  information  - specular  images,  Lambertian  images, 
surface  orientations,  and  3-D  depth  map  - all  derived  from  the  original  raw  images  will 
be  used  to  inspect  or  recognize  underlying  specular  objects  most  effectively  as  we  will 
discuss  in  the  following  chapters. 

3.2  Highlight  Separation  and  Surface  Shape  of  3-D  Specular  Objects 

In  this  section,  we  will  discuss  a computational  approach  for  separating  probable 
highlights  and  obtaining  surface  orientations  of  3-D  specular  objects.  We  will  begin  by 
discussing  how  to  obtain  surface  orientations  for  a Lambertian  surface  using  three 
monochromatic  images.  Next,  we  will  use  the  simplified  Torrance-Sparrow  model  to 
develop  a method  for  obtaining  surface  orientations  for  a specular  surface  using  three 
monochromatic  images.  We  then  present  the  approach  for  separating  probable  highlights 
based  on  the  normal  vector  equalization  method.  We  will  also  discuss  how  to  utilize  this 
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method  to  obtain  surface  orientations  of  specular  objects.  Then,  we  will  discuss  the 
problem  of  practical  implementation  of  the  highlight  separation  algorithm.  After  that,  a 
depth  recovery  method  based  on  a line-integration  technique  will  be  discussed. 

3.2.1  Surface  Orientation  for  Lambertian  Surfaces 

Assuming  a Lambertian  surface,  surface  orientation  can  be  determined  using  three 
images  obtained  from  three  different  positions  of  a light  source.  This  is  accomplished  by 
beginning  with  the  imaging  equation  for  Lambertian  reflection  model  defined  in  (2.6). 
From  (2.6),  we  have: 


I„(x,y)  - RS-N  = R(Six,S.y,Si2)iN^yyNz)  , *=1,2,3  (3.1) 

where  N denotes  surface  normal  vector,  and  S,  and  IJx,y)  denote  the  ith  light  source 
vector  and  the  image  intensity  produced  by  the  ith  light  source.  Since  (3.1)  contains  three 
unknowns,  i.e.  a total  of  three  sources  and  associated  intensity  values  are 

usually  used  to  compute  the  surface  normal  vectors  at  each  point,  (jt,y)  [Ike81a,  Col82]. 
Given  the  sources  Slt  S2,  and  S3  along  with  the  intensity  values  Iu,  Il2,  and  I,3  at  (x,y),  a 
set  of  simultaneous  equations  can  be  formed  to  solve  for  N: 


I.  = 


r \ 
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\l3J 


= R 
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S,  S,  S,  N 
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S0  S,  S,  N 
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S , S,  S,  N 

V 3jr  3zA  *J 


(3.2) 


If  we  use  S to  symbolize  the  source  matrix,  I,  the  Lambertian  intensity  vector  at  (x,y)  and 
N the  normal  vector  at  (x,y),  (3.2)  becomes: 
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I,  = R SN  (3-3) 

To  solve  for  surface  normal  vector,  if  S is  a nonsingular  matrix,  (3.3)  can  be  written  as: 

N = — S-'I.  (3-4) 

R ~ ' 

The  conditions  for  the  nonsingularity  of  5 is  (1)  the  row  of  S are  linearly 
independent  or  (2)  the  column  of  5 are  linearly  independent  [Cul72].  When  Sv  S2,  and 
S3  are  not  all  in  the  same  plane,  they  are  linearly  independent  and  therefore  S is 
nonsingular.  This  means  that  the  light  sources  must  be  installed  so  that  their  vectors 
constitute  some  volume  in  the  imaging  space. 

In  Fig.  3.1,  we  illustrate  the  imaging  coordinate  system  including  light  sources  and 
a camera.  Three  azimuthal  angles  (0a)  of  approximately  0°,  120°,  and  240°  are  chosen  for 
three  source  locations  S,,  i=  1,2,3.  An  elevation  angle  (0e)  of  approximately  75°  is  chosen 
for  the  experiments  with  real  objects  and  85°  is  chosen  for  the  experiments  with 
synthesized  images,  respectively.  Obviously,  these  settings  of  light  sources  satisfy  the 
nonsingularity  condition  for  S mentioned  above.  A camera  is  located  at  the  direct  top  of 
an  object  so  that  the  viewing  vector  V becomes  (0,0,1). 

3.2.2  Local  Shape  from  Specularity 

Based  on  the  discussions  given  in  Section  2.2,  we  describe  the  simplified 
Torrance-Sparrow  model  for  the  ith  light  source  as  follows  [Hea88a]: 


camera 


u 


Fig.  3.1.  Imaging  coordinate  system. 
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r = Kex p( 


(3.5) 


m 


2 


where  K and  m are  defined  in  (2.11),  Isi  is  the  ith  image  intensity  produced  by  the 
specular  reflection  mode,  and  a,  is  the  angle  between  N and  H{  which  is  associated  to 
the  ith  light  source  S,  as  shown  in  Fig.  2.1.  For  specular  surfaces,  surface  normal  vectors 
around  highlights  can  also  be  calculated  with  three  images  obtained  from  three  different 
positions  of  a light  source.  From  (3.5),  we  have: 


If  we  use  symbols  IsI,  ls2,  and  Is3  for  three  specular  images  and  Ht,  H2,  and  H}  for  the 
corresponding  bisectors,  we  have  from  (3.7): 


(3.6) 


We  arrange  (3.6)  in  terms  of  unknown  variables,  N and  have: 


HtN  = cos[/n  ln(-^)]  , i=l,2,3 

N 7« 


(3.7) 


(3.8.a) 


where 
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and 
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h=h2  (3,8c) 

\ ) 

Here,  H is  assumed  to  be  a nonsingular  matrix.  It  is  known  that  H is  always  nonsingular 
if  and  only  if  5 is  nonsingular. 

The  surface  orientations  for  a specular  surface  represented  by  (3.8.a)  will  be 
combined  together  in  the  next  section  with  those  for  a Lambertian  surface  represented  by 
(3.4)  to  separate  probable  highlights  from  the  original  images  and  produce  the  images  of 
two  reflection  components,  i.e.  Lambertian  and  specular. 

3.2.3  Lambertian  and  Specular  Components  Separation 


As  discussed  earlier,  a general  reflectance  model  considering  both  the  Lambertian 
reflection  and  the  specular  reflection  can  be  described  as  the  weighted  sum  of  the 
reflection  components.  That  is. 
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L(x,y)  = ylu(x,y)+(l-y)l3.(x,y)  = I^X^y)  , *=1,2,3  (3-9.a) 

where 

I'Xxj)  = ylh(x,y)  , *=1,2,3  (3-9.b) 

and 

i;  = (1  -y )IJ.x,y)  , <=1,2,3  (3.9.c) 

In  (3.9),  f i,(x,y)  represents  the  ith  weighted  intensity  of  the  Lambertian  component  and 
fs/(x,y)  represents  the  ith  weighted  intensity  of  the  specular  component.  Combining  (3.9.b) 
with  (3.4),  we  have: 

N = -Is-'/,*  (3.10.a) 

R'~ 

where 

R*  = yR  (3.10.b) 

and  similarly  from  (3.9.c)  and  (3.8),  we  have: 

N = Hxg * (3.11. a) 


where 
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ln(£l)  ] 
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cos[m 

me'*)  ] 

* J 

(3.1 l.b) 


K * = (l-y)A'  (311c) 

For  a given  surface,  surface  normal  vectors  are  unique.  Therefore,  the  surface 
normal  vectors  obtained  from  each  component  should  be  the  same;  hence  from  (3.10.a) 
and  (3.11. a),  we  have: 


— 5-7,*  = H'g'  (3.12) 

R.~  — 

Applying  the  relation,  /*,  = /-  fs  derived  from  (3.9.a)  to  (3.12),  we  have: 

G(0  a H 'g'+ = 0 (3.13) 

R'~ 

(3.13)  represents  three  nonlinear  equations  which  have  three  unknowns,  namely  T sl,  /*j2, 
and  I* s3.  (3.13)  can  be  solved  by  numerical  analysis  using  Newton’s  method  in  the 
following  iterative  equation  [Tay73]. 

where  f,(n)  represents  the  value  of  l\  after  the  nth  iteration  and  is  a 3X3  Jacobian 
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matrix  whose  (u,v)th  element  is  given  by  dGJdl*„(n),  where  w,v=l,2,3.  We  define  the 
following  dummy  matrices  to  simplify  the  description  of  the  Jacobian  matrix. 


C = 


f \ 


C3 

V J 


= H 


-i 


(3. 15. a) 


D = 


/ \ 

Dt 

I>2 


With  these  matrices,  the  Jacobian  matrix  is  given  by: 


(3.15.b) 


J = 


'CUX(0*DU 

c2iMO*d„ 

c,MO>d„ 


caMO+Dn  c„HO*Da 
c2,ui;,)*d21 
c„uo+d12  c„HO*D„ 


where  ty/*„),  j=l,2,3,  is  given  by: 


(3.16) 


m sin[m 
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(3.17) 


In  order  to  solve  (3.14)  for  a set  of  surface  parameters  {R* ,K* ,m)  or  another 
set  of  parameters  \R,K,y,m)  must  be  provided  into  (3.15.b),  (3.16),  and  (3.17).  These 
parameters  can  be  measured  using  a sample  of  the  objects  under  application  by  employing 
the  method  discussed  in  Chapter  4. 
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After  we  find  the  weighted  specular  component  by  solving  (3.14),  the  weighted 
Lambertian  component  can  be  obtained  by  the  following  simple  equation  derived  from 
(3.9. a). 


From  (3.9.b)  and  (3.9.c),  the  non-weighted  Lambertian  and  specular  components  are  given 
by: 


The  surface  normal  vectors  can  be  calculated  by  (3.4)  using  the  non-weighted 
Lambertian  component  or  by  (3. 10. a)  using  the  weighted  Lambertian  component.  The 
physical  significance  of  these  non-weighted  Lambertian  and  specular  components  is  that 
they  represent  fictitious  image  intensities  which  can  be  obtained  if  a given  surface  is 
changed  into  a perfectly  Lambertian  surface  or  a perfectly  specular  surface,  respectively, 
without  any  change  of  other  imaging  parameters  such  as  imaging  geometry,  light  source 
intensity,  and  camera  sensitivity. 


J,*(x,y)  = /(x,y) -/,*(*, y) 


(3.18) 


(3.19.a) 


and 


(3.19.b) 
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3.2.4  Highlight  Separation  Algorithm 

The  processes  of  separating  highlights  discussed  in  the  preceding  section  can  be 
informally  described  as  follows: 

Algorithm  1 ; Highlight  Separation 

1.  Input:  I=[I1J2J3],  K*,  R*,  m,  S,  V 

2.  Create:  H,  C,  D 

3.  Scan:  images  from  left  to  right  and  top  to  bottom 

3.1  if(/,=0  for  all  *=1,2,3)  then  set  /*,=0  for  all  *=1,2,3  and  goto  step  3 
else 

3.2  let  I* 3i(o) =K* /2  for  all  *=1,2,3 

3.3  do  n=lflmax 

3.3.1  determine  g*(f,lnl)  using  (3.11.b) 

3.3.2  determine  G(tslllf)  using  (3.13) 

3.3.3  if(|G//*,w)|^e,  for  all  *=1,2,3)  goto  step  3 

3.3.4  determine  using  (3.17) 

3.3.5  determine  J_  and  £l  using  (3.16)  and  matrix  inversion 

3.3.6  determine  I*sl„+Il  using  (3.14) 

3.3.7  if(tsi(n+I)<0)  then  let  fsi(n+1)=l  for  *=1,2,3 

3.3.8  if(rsi(n^)  then  let  l\i(n+l}=K*  for  *=1,2,3 

3.3.9  if(  1 1* ,i(n+i) \^2  for  all  *=1,2,3)  then  goto  step  3 


end  do 
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This  algorithm  is  mainly  composed  of  three  steps:  They  are  (1)  an  input  step  for 
providing  the  algorithm  with  basic  parameters  representing  imaging  geometry  and  optical 
characteristics  of  a given  surface,  (2)  a calculation  step  for  deriving  some  quantities  from 
the  basic  parameters,  and  (3)  an  iterative  calculation  step  for  finding  specular  reflection 
component  for  all  pixels  in  the  image.  Since  the  first  two  steps  are  simple  and 
straightforward,  we  will  focus  on  the  third  one  in  the  discussion  here. 

In  the  third  iterative  processing  step,  four  stopping  conditions  [Van78]  are 
employed  to  prevent  infinite  looping  and  also  to  keep  the  computation  time  as  minimal 
as  possible.  They  are  (1)  Step  3.1  that  means  the  perceived  images  are  all  for  extremely 
dark  area  such  as  background  or  self-shading  area,  where  highlight  reflection  can  never 
occur;  (2)  Nmwc  employed  in  Step  3.3,  which  represents  the  number  of  maximum  allowable 
iterations;  (3)  e,  employed  in  Step  3.3.3,  which  means  a threshold  that  represents  how 
ft(n}  must  be  close  to  exact  solutions  of  (3.13)  in  order  for  the  algorithm  to  stop;  and  (4) 
e2  employed  in  Step  3.3.9,  which  means  a threshold  that  represents  how  closely  the 
difference,  converges  to  zero.  These  thresholds  are  determined  by  the 

accuracy  of  results  required  in  the  applications.  For  instance,  if  only  integer  resolution  is 
sufficient  for  integer  1 may  be  chosen  for  e2.  To  determine  N^,  the  convergence 
speed  of  the  algorithm  should  be  investigated.  As  denoted  in  Fig.  3.2,  the  algorithm 
shows  quite  fast  convergence  characteristics.  After  performing  a number  of  experiments, 
we  have  found  that  Nma=  5 can  provide  satisfactory  results  for  integer  resolution  of  I*,,  e, 
is  necessary  to  determine  how  closely  f,(H)  converges  to  exact  solution  t,  which  satisfies 
(3.13).  In  the  highlight  separation  algorithm,  G(f,lnj)  in  Step  3.3.2  must  be  calculated  with 
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Fig.  3.2.  Convergence  of  (a)  Before  the  1st  iteration  (,\t s,-f sl(0}\).  (b)  After  the  1st 
iteration  (|/*w-/*ww  |)  (*  denotes  the  number  greater  than  9).  (c)  After  the  2nd  iteration 

( sl'f  sl(2}\)‘ 
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the  resolution  of  a real  number.  Therefore  e , must  be  real  and  a smaller  value  of  it  makes 
the  algorithm  run  until  more  accurate  results  are  obtained.  As  in  the  case  of  Nmax,  e , has 
been  experimentally  determined  for  the  optimum  value  between  accuracy  and  processing 
time.  In  the  following  experiments,  we  employed  the  stopping  parameters  of  Nmax= 5, 
e,=0.1,  and  e2=l. 

In  Step  3.2,  initial  values  for  the  specular  component  are  given  to  the  algorithm. 
They  are  determined  as  lC/2  because  this  value  is  the  middle  of  the  whole  dynamic  range 
of  the  specular  component.  We  can  describe  this  initialization  scheme  as  fixed 
initialization,  since  the  initial  values  are  all  fixed  regardless  of  the  location  of  pixels. 
Another  possible  choice  for  the  initial  values  is  to  employ  the  solutions  derived  for  the 
previous  pixel  location  as  the  initial  values  of  present  pixel  location.  In  other  words,  if 
the  algorithm  is  now  working  on  the  pixel  location  (x,y)  and  fjx-l,y)  is  available  for  the 
previous  location,  then  this  adaptive  initialization  scheme  let  t ^(0}(x,y)-t  Jx-1  ,y)  for  all 
i=l,2, 3.  This  adaptive  initialization  scheme  utilizes  the  correlation  between  the  nearest 
pixels.  The  correlation  means  that  the  value  of  a pixel  is  constrained  by  the  value  of  the 
nearest  pixels.  Many  algorithms  for  image  data  compression  are  based  on  this  correlation 
between  the  pixels  which  is  often  referred  to  as  redundancy  [Pra79].  We  performed 
experiments  to  study  the  convergence  speed  of  the  algorithm,  when  either  the  fixed 
initialization  or  the  adapted  initialization  is  employed.  The  results  showed  that  no 
significant  change  occurred  in  the  overall  processing  speed  of  the  algorithm.  This  is 
because  the  convergence  speed  of  the  algorithm  is  fast  enough  so  that  the  small  difference 
in  the  initialization  values  is  not  so  influential  to  the  overall  processing  speed. 


48 


In  Fig.  3.2,  differences  between  exact  solution,  fsl  and  approximate  solution  after 
the  nth  iteration,  flI(n}  are  illustrated  for  three  cases  of  n= 0,  1,  and  2.  These  results  have 
been  obtained  using  synthesized  sphere  images  generated  from  the  reflection  models 
explained  in  (3.5),  (3.6),  and  (3.9).  The  convergence  characteristics  illustrate  that  the 
algorithm  can  find  almost  an  exact  solution  only  after  two  iterations.  Details  regarding 
the  simulation  will  be  discussed  in  Section  3.4. 

While  the  algorithm  runs,  it  may  temporarily  generate  the  value  of  which 
may  be  less  than  zero  for  extremely  dark  areas  such  as  background  or  self  shading  areas 
where  the  incidence  angles  are  greater  than  90°  or  may  be  greater  than  K*  for  the  areas 
around  highlights.  In  order  to  ensure  that  (3.1 1 .b)  and  (3.17)  can  always  be  defined  even 
in  these  extraordinary  cases,  l*si(n+Ij  is  applied  to  the  hard  limitation  processes  in  Step 
3.3.7  and  3.3.8.  As  a result,  fsi(n +1)  is  always  confined  in  the  range  of  0<J*sl(n+1)<iK‘  for  all 
(=1,2,3. 


3.2.5  Surface  Reconstruction  from  Surface  Normal  Vectors 

Using  the  approach  discussed  in  the  previous  sections,  we  can  obtain  images  of 
separated  reflection  components  and  surface  normal  vectors.  Although  surface  normal 
vectors  have  sufficient  information  for  the  description  of  3-D  shape  of  objects  using,  for 
instance,  an  intrinsic  image  [Bar78],  it  is  often  interesting  and  important  to  recover  the 
relative  depth  of  the  surface  which  is  often  referred  to  as  depth  map. 

In  this  section,  we  will  discuss  the  approach  for  depth  recovery  based  on  Coleman 
and  Jain’s  work  [Col82]  and  Wu  and  Li’s  work  [Wu88].  In  Coleman  and  Jain’s  work,  the 
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depth  is  estimated  by  considering  the  normal  vectors  at  two  adjacent  points  of  some  point, 
computing  the  average  tangent  through  the  points,  and  interpolating  them  to  find  the 
relative  depths.  In  Wu  and  Li’s  work,  an  arbitrary  depth  is  first  assigned  for  a point  in 
an  image,  then  path-independent  line  integrals  are  computed  to  obtain  the  relative  depths 
at  every  point  in  the  image. 

The  depth  recovery  method  employed  in  this  dissertation  locally  follows  the  idea 
of  Coleman  and  Jain’s  approach  by  interpolating  the  tangent  of  two  points  which  are  in 
an  integration  path  and  globally  follows  the  idea  of  Wu  and  Li’s  approach  by  employing 
the  line-integral  based  approach. 

Mathematically,  the  problem  of  depth  recovery  can  be  described  as  finding  the 
surface  of  an  object  using  its  normal  vectors.  For  a surface  given  by  z=f(x,y),  the  surface 
normal  vectors  can  be  described  using  a gradient  of  a scalar  surface  defined  by, 
F(x,y)£f(x,y)-z= 0,  as  discussed  in  Section  2.1.  From  (2.4),  p and  q are  given  by. 


N 


(3.20.a) 


P = - 


* 


N 


and 


N 


q = - 


y 


(3.20.b) 


N 


From  the  definitions  of  p and  q and  (3.20.a),  we  have, 
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dz  Az|x  Nx 

P " Bx  ~ ~ W 

(3.21. a) 

and  similarly, 

4 = * . H . 

By  Ay  Nz 

(3.21. b) 

where  A z is  the 

depth  difference  between  {x0,y0)  and  its  adjacent  point  (x,,y0)  and  A z\y 

denotes  the  depth 

difference  between  (x0,y0)  and  (jc0,y7),  as  illustrated  in  Fig.  3.3.  That  is, 

)-z(^0,yo)  » ^1,  = *(Wi)-*(Wo) 

(3. 22.  a) 

ii 

J-e 

**i 

o* 

(3.22.b) 

$ 

II 

1 

(3.22.c) 

In  many  cases,  both  Ax  and  Ay  are  regarded  as  a unit  pixel  distance.  If  we  solve  (3. 21. a) 
and  (3.21.b)  for  Az\x  and  Az|r  respectively  and  interpolate  Nx,  Ny,  and  Nz  using  a pair 
of  them  associated  to  Ax  or  Ay,  we  have  the  following  equation  for  the  path  denoted  as 
C,  in  Fig.  3.3. 


( ^Vo)+iV>i^o)  ^ 


Az|  « 


N 

x 

~N 


lAx  = - 


NMo^+NMvyo) 


Ax 


(3.23. a) 


Similarly  from  (3.21.b),  we  have  the  following  equation  for  the  path  C,. 


Fig.  3.3.  Rectangular  grid  in  an  image. 
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N 

Azl  = -—Ay  ~ - 
,y  N 

Z 

z(xt,y,),  surface  depth  at  (jt;,y7)  can  be  obtained  by  repeating  (3.23)  along  the  integration 
paths  denoted  as  C,-C4  or  C2-C3  in  Fig.  3.3.  Given  z(x0,y0),  for  instance,  if  we  use  the 
integration  path  C,-C4  for  solving  z(Xj,y,),  we  have  from  (3.23), 


y^O’^CK  + 


A^0,y0)+/V(*0,y,) 


(3.23.b) 


z(*iO'i)  lc,-c4 


Nx(x0,y0)+Nx(xvy0) 
NfayJ+Nfay 0) 


"M, 


Similarly,  if  we  use  the  path  Q-Cj,  we  have. 


(3. 24.  a) 


zC*iJi)lc1-c, 


z(*o0’o)-0'i-:yo) 


Ny(x0,y0)^Ny(x0,y,) 

NMo^o>+NMo^i) 


,v  x.,NSxo>yJ+NMvyJ 

1 Nz(x0,y , ) +Nz(x , ,y , ) 


(3.24.b) 


In  order  to  obtain  the  depth  map  for  the  whole  image  denoted  as  z(x,y),  we  can 
repeat  the  integration  processes  discussed  above  for  the  entire  image.  The  depth  at  the 
starting  point  z(x0,y0)  is  first  assumed  and  relative  depths  compared  to  z(x0,y0)  are  calculat- 
ed by  repeating  the  integration  processes.  The  starting  point  is  determined  as  the  center 
of  an  image.  The  relative  depth  at  every  pixel  is  finally  determined  by  averaging  the  two 
depths  obtained  from  each  integration  path.  That  is, 
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z(x,y)  = I{^)|Ci.C4+z(^)|Ci.c>}  (3.25) 

3.3  Experiments  with  Real  Specular  Objects 
3.3.1  Experimental  Samples 

In  this  dissertation,  we  have  selected  several  spherical  samples  to  conduct 
experiments  for  (1)  separating  highlights,  (2)  obtaining  3-D  shape  descriptions,  and  (3) 
measuring  reflectance  parameters.  Of  these  experiments,  measurements  of  reflectance 
parameters  are  discussed  in  Chapter  4.  In  this  chapter,  we  will  only  adopt  the 
measurement  results  to  conduct  the  first  two  experiments.  The  samples  are  listed  as 
follows: 

(1)  Sphere  A:  which  is  made  from  highly  lustrous  plastic. 

(2)  Sphere  B:  which  is  made  from  common  plastic  with  medium  range  of  specularity. 

(3)  Sphere  C:  which  is  made  from  highly  diffusive  sponge. 

These  samples  were  selected  as  representative  of  the  gamut  of  objects,  ranging  from 
highly  specular  to  near  Lambertian. 

The  other  samples  which  do  not  have  spherical  shape  are: 

(1)  Crayon:  which  is  made  from  plastic -like  coloring  matter. 

(2)  Transistor:  which  is  made  from  metal. 

(3)  Button:  which  is  made  from  shiny  plastic. 

(4)  Bulb:  which  is  made  from  glass  coated  with  white  paint. 

(5)  Capsule:  which  is  coated  with  gelatine. 
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The  reflectance  parameters  of  these  non-spherical  samples  are  also  provided  from  the 
experimental  results  discussed  in  Chapter  4.  In  Table  3.1,  the  measured  values  of  the 
reflectance  parameters  are  presented  for  those  samples. 

3.3.2  Setup  for  Experimentation 

In  Fig.  3.4(a),  we  illustrate  the  hardware  structure  of  the  PC -based  image 
processing  system  in  which  the  algorithms  discussed  in  the  previous  sections  are 
implemented.  The  frame  grabber,  DT-2853  developed  from  Data  Translation,  is  installed 
in  a PC  (Personal  Computer)-AT  system.  The  frame  grabber  captures  an  image  of  up  to 
512X512X8-bit  resolution,  stores  the  images  in  one  of  two  on-board  memory  buffers,  and 
displays  the  image  in  RGB  false  color  or  monochrome  at  a rate  of  30  image  frames  per 
second.  A camera  developed  from  Cohu  is  used  to  take  images  and  provide  the  frame 
grabber  with  high  resolution  image  signal.  For  displaying  processed  images,  an  analog 
RGB  monitor  is  connected  to  the  frame  grabber. 

A device  holding  the  camera  and  light  sources  has  been  built  to  control  the 
elevation  angles  and  the  azimuthal  angles  of  the  light  sources  over  the  object  under 
experiment.  In  Fig.  3.4(b),  we  illustrate  the  holding  device. 

Before  we  start  actual  experimentation,  we  need  to  calibrate  the  sensitivity  of  the 
camera  and  the  locations  and  the  intensity  of  the  light  sources.  The  sensitivity  of  the 
camera  can  be  adjusted  using  the  exposure  rate  of  the  lens  mounted  on  the  camera.  The 
locations  of  the  light  sources  are  controlled  by  changing  their  azimuthal  and  elevation 
angles.  The  intensity  of  each  light  source  is  adjusted  by  changing  the  opening  area  of  a 
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Table  3.1.  The  reflectance  parameters  of  the  samples. 


Sample 

R' 

/C 

m 

Sphere  A 

92.4 

166.2 

0.032 

Sphere  B 

119.0 

121.6 

0.085 

Sphere  C 

178.1 

7.5 

0.550 

Crayon 

104.9 

121.8 

0.07f 

Transistor 

85.6 

169.2 

0.045f 

Button 

104.4 

150.6 

0.04f 

Bulb 

152.4 

95.7 

0.06f 

Capsule 

127.3 

104.3 

0.05f 

t The  values  of  m for  these  non-spherical  samples  are  estimated  from  the  measurement 
results  for  the  spherical  samples. 
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(b) 

Fig.  3.4.  Image  acquisition  and  processing  system,  (a)  PC-based  image  processing  system, 
(b)  Holding  device  for  image  acquisition. 
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slit  which  covers  the  light  source.  These  factors  are  all  adjusted  in  combination  so  that 
the  resultant  images  make  use  of  the  full  available  dynamic  range  of  the  image  processing 
system. 

In  order  to  measure  the  source  vectors,  we  used  Sphere  A and  followed  the 
method  described  in  Chapter  4.  In  Table  3.2,  we  present  the  measured  source  vectors 
adopted  from  the  experimental  results  given  in  Chapter  4.  They  are  described  in  terms  of 
the  azimuthal  angles  (0a)  and  the  elevation  angles  (9e)  defined  in  Fig.  3.1.  These  data  are 
all  provided  to  the  highlight  separation  algorithm  discussed  in  Section  3.2.4. 

3.3.3  Experimental  Results 

In  Fig.  3.5,  we  present  the  experimental  results  for  the  spherical  samples.  They 
include  (1)  highlight  separation  results  for  Sphere  A,  B,  and  C,  (2)  surface  normal  vectors 
represented  by  the  needle  map,  and  (3)  3-D  depth  map  derived  from  the  normal  vectors 
using  the  depth  recovery  method.  The  needle  map  and  3-D  depth  map  presented  in  Fig. 
3.5  are  obtained  from  sphere  B.  For  Sphere  A and  C,  similar  results  have  been  obtained. 

For  non-spherical  samples,  we  present  the  experimental  results  on  the  Crayon  in 
Fig.  3.6,  on  the  Transistor  in  Fig.  3.7,  on  the  Button  in  Fig.  3.8,  on  the  Bulb  in  Fig.  3.9, 
and  on  the  Capsule  in  Fig.  3.10,  respectively.  Using  the  reflectance  parameters  and  source 
vectors  adopted  from  the  experimental  results  given  in  Chapter  4,  we  have  obtained  quite 
successful  results  for  these  non-spherical  samples. 
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Table  3.2.  Measured  source  vectors. 


Sample 

0ai.  0a2>  0a3  (degree) 

9ai>  9.2*  0.3  (degree) 

Spherical 

2.49,  122.9,  235.6 

52.2,  56.9,  52.2 

Non-spherical 

5.7,  125.5,  258.7 

73.7,  76.1,  73.5 
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(a) 


(b) 


(c) 


Fig.  3.5.  Experimental  results  on  the  spherical  samples.  The  images  in  the  left  column 
denote  the  original  raw  unages,  those  in  the  center  column  denote  the  specular  images, 
and  those  in  the  right  column  denote  the  Lambertian  images,  (a)  Sphere  A.  (b)  Sphere  B. 
(c)  Sphere  C.  (d)  Needle  map.  (e)  Depth  map. 
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Fig.  3.5.  - Continued. 
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Fig.  3.6.  Experimental  results  on  the  Crayon,  (a)  Highlight  separation  results:  The  images  in  the  left  column  denote  the 
original  raw  images,  those  in  the  center  denote  the  specular  images,  and  those  in  the  right  denote  the  Lambertian 
images,  (b)  Needle  map.  (c)  Depth  map. 
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Fig.  3.6.  - Continued. 
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Fig.  3.7.  Experimental  results  on  the  Transistor,  (a)  Highlight  separation  results:  The  images  in  the  left  column  denote 
the  original  raw  images,  those  in  the  center  denote  the  specular  images,  and  those  in  the  right  denote  the  Lambertian 
images,  (b)  Needle  map.  (c)  Depth  map. 
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Fig.  3.7.  - Continued. 
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Fig.  3.8.  Experimental  results  on  the  Button,  (a)  Highlight  separation  results:  The  images  in  the  left  column  denote  the 
original  raw  images,  those  in  the  center  denote  the  specular  images,  and  those  in  the  right  denote  the  Lambertian 
images,  (b)  Needle  map.  (c)  Depth  map. 
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Fig.  3.8.  - Continued. 
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Fig.  3.9.  Experimental  results  on  the  Bulb,  (a)  Highlight  separation  results:  The  images  in  the  left  column  denote  the 
original  raw  images,  those  in  the  center  denote  the  specular  images,  and  those  in  the  right  denote  the  Lambertian 
images,  (b)  Needle  map.  (c)  Depth  map. 
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Fig.  3.9.  - Continued. 
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Fig.  3.10.  Experimental  results  on  the  Capsule,  (a)  Highlight  separation  results:  The  images  in  the  left  column  denote 
the  original  raw  images,  those  in  the  center  denote  the  specular  images,  and  those  in  the  right  denote  the  Lambertian 
images,  (b)  Needle  map.  (c)  Depth  map. 
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Fig.  3.10.  - Continued. 
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3.4  Simulation  with  Synthesized  Images 

Computer  simulation  using  the  shape  of  a hemisphere  has  been  performed  (1)  to 
study  the  system  operation  including  the  convergence  characteristics  of  the  algorithm 
discussed  in  Section  3.2.4  and  (2)  to  find  sensitivities  of  (a)  surface  shape  z(x,y),  (b) 
Lambertian  components  and  (c)  specular  components  fjx,y)  with  respect  to  the 

change  of  surface  parameters  - y and  m.  We  will  first  describe  the  simulation  procedures 
and  then  discuss  the  results  of  error  analysis. 

3.4.1  Simulation  Procedures 

The  whole  procedures  of  computer  simulation  are  presented  in  Fig.  3.11.  First  of 
all,  the  shape  of  a hemisphere  is  modelled  using  its  normal  vectors  which  are  given  by 
N=(x/r,y/r,z/r)  and/or  directly  using  its  surface  function  given  by  z(x,y)=-(r2-xl-y2)1'2,  where 
r denotes  the  radius  of  the  hemisphere.  To  generate  synthetic  images  which  will  be 
provided  into  the  simulation  processes,  we  use  the  general  reflectance  model  discussed 
in  Section  2.2.  Imaging  parameters  V and  S are  incorporated  into  (3.1),  (3.5),  and  (3.9) 
along  with  surface  parameters  ya,  m0,  K,  and  R to  synthesize  a set  of  original  images  I 
which  contain  the  Lambertian  component  (/*')  and  the  specular  component  (/*').  In 
generating  the  synthetic  images,  three  representative  values  of  surface  parameters  which 
describe  a highly  specular  object  (y=0.3,  mo=0.3),  medium  specular  object  (yo=0.5,  m = 
0,5),  and  near  Lambertian  object  (yo=0.7,  m0= 0.7)  have  been  selected  to  simulate  the 
system  capability  for  objects  made  with  various  kinds  of  materials  ranging  from  highly 
specular  surfaces  to  near  Lambertian  surfaces.  The  values  of  K and  R have  been 
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Fig.  3.11.  Simulation  procedures. 
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determined  as  255,  although  they  are  not  necessarily  the  same,  since  the  digitizer  used  in 
the  experiments  with  real  objects  has  8 -bit  resolution  and  the  light  intensity  has  been 
adjusted  so  that  the  brightest  part  of  the  objects  is  almost  saturated  in  the  images.  As  a 
consequence  of  these  settings,  we  can  retain  consistency  between  simulation  and 
experiments. 

Secondly,  the  highlight  separation  algorithm  runs  to  obtain  a set  of  Lambertian 
images  t,  and  a set  of  specular  images  T,  from  the  original  images  /.  In  this  stage, 
surface  parameters  denoted  as  yd  and  md  are  used.  For  the  first  simulation  purpose 
discussed  above,  the  same  surface  parameters  as  used  in  the  image  generation  are 
employed  and  the  convergence  characteristics  are  studied.  In  other  words,  we  assume  such 
ideal  conditions  that  we  know  the  exact  values  of  surface  parameters  a priori  by  setting 
yd=Y0  and  md=m0.  Under  these  ideal  conditions,  we  also  performed  the  experiments  on 
highlight  separation,  needle  map  generation,  and  depth  map  recovery  using  the  images 
synthesized  for  those  three  representative  values  of  yc  and  m0\  (1)  yo=0.3,  m= 0.3,  (2) 
yo=0.5,  m=  0,5,  and  (3)  yo=0.7,  m=0.1.  In  Fig.  3.12,  we  present  the  experimental  results 
only  for  yo=0.5  and  m= 0.5.  The  reconstructed  surface  i(x,y)  is  also  illustrated  in  Fig.  3.12 
along  with  the  original  shape  z(x,y)  and  the  absolute  error  between  two  shapes,  i.e. 
\z(x,y)-i(x,y)\.  They  represent  the  shape  of  a hemisphere  quite  accurately. 

3.4.2  Error  Analysis 

The  reason  we  have  obtained  accurate  reproduction  of  the  hemisphere  in  the 
previous  simulation  work  is  mainly  that  we  have  assumed  ideal  conditions  to  use  exactly 
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Fig.  3.12.  Experimental  results  on  synthesized  images  (0a=O°,  120°,  240°,  0e=85°, 
Yo=Yd=0-5,  m0=my= 0.5).  (a)  Highlight  separation  results:  The  images  in  the  left  column 
denote  the  original  raw  images,  those  in  the  center  denote  the  specular  images,  and  those 
in  the  right  denote  the  Lambertian  images,  (b)  Needle  map.  (c)  Original  shape,  z(x,y).  (d) 
Reconstructed  shape,  i(x,y).  (e)  Absolute  error,  \z(x,y)-2(x,y)\. 
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Fig.  3.12.  - Continued. 
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the  same  surface  parameters  in  highlight  separation  and  normal  vector  derivation  stages 
as  those  used  in  image  generation.  In  real  applications,  however,  it  is  impossible  to  find 
the  values  of  yc  and  m0  and  we  must  tolerate  some  error  in  estimating  the  surface 
parameters  of  real  objects  even  when  the  parameter  measurement  method  discussed  in 
Chapter  4 is  employed.  Therefore,  the  second  simulation  work  has  been  performed  to 
investigate  how  sensitively  (1)  reconstructed  surface  shape,  (2)  separated  Lambertian 
components,  and  (3)  separated  specular  components  are  affected  by  the  surface  parameter 
errors  compared  to  the  originals.  For  this  purpose,  we  intentionally  use  different  surface 
parameters  in  highlight  separation  and  normal  vector  derivation  stages  from  those  used 
in  original  image  generation  to  simulate  surface  parameter  errors  which  cannot  be  avoided 
in  real  applications.  In  other  words,  we  set  yd*y0  and  as  denoted  in  the  diagram 

shown  in  Fig.  3.11  and  investigate  (1)  error  of  reconstructed  surface  Ez,  (2)  error  of 
Lambertian  components  E„  and  (3)  error  of  specular  components  Es  with  respect  to  the 
surface  parameter  errors  Ay=|yd-yc|  and  Am=\mrm0\.  Mathematically,  the  errors  are 
defined  as: 
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where  z(x,y),  f ,,{x,y),  and  f Jx,y)  denote  the  original  shape  of  the  hemisphere  used  to 
synthesize  the  original  images,  the  original  Lambertian  images,  and  the  original  specular 
images,  respectively.  i(x,y),  f Jx,y),  and  tjx.y)  in  (3.26)  denote  the  reconstructed  shape 
of  the  hemisphere  and  estimates  of  Lambertian  and  specular  images  obtained  after 
performing  surface  reconstruction  and  highlight  separation  processes.  M denotes  the 
number  of  pixels  within  the  sphere  and  Mh  denotes  that  of  pixels  with  in  a highlight  area. 
In  Chapter  4,  we  will  discuss  how  to  determine  the  highlight  area. 

In  (3.26),  Ez  describes  the  global  relative  error  of  the  reconstructed  surface 
averaged  for  overall  sphere  area.  In  other  words,  Ez  represents  how  accurately  we  can 
recover  the  original  shape  when  we  allow  the  surface  parameter  measurement  errors  of 
Ay  and  Am.  E,  and  Es  describe  the  average  relative  errors  of  the  estimated  Lambertian 
component  and  specular  component,  respectively.  They  represent  how  accurately  we  can 
estimate  the  original  Lambertian  component  and  specular  component  in  the  global  sense. 
Note  that  the  summation  region  in  (3.26.c)  is  changed  from  the  whole  sphere  area 
employed  in  (3. 26. a)  and  (3.26.b)  into  the  highlight  area  in  which  specular  components 
have  an  appreciable  amount. 

For  three  representative  cases  of  yc  and  m0  mentioned  above,  plots  of  (3.26.a), 
(3.26.b),  and  (3.26.c)  are  presented  in  Fig.  3.13,  3.14,  and  3.15,  respectively.  From  the 
plots  in  Fig.  3.13,  we  have  found  that  if  the  relative  parameter  errors  (Ay/y0,  Am/m0) 
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remain  within  ±50%,  the  reconstructed  surface  error  Et  is  within  35%  and  if  the  parameter 
errors  remain  within  ±20%,  then  the  reconstructed  surface  error  can  be  confined  within 
25%  in  the  global  sense.  From  the  plots  in  Fig.  3.14,  we  also  found  that  the  averaged 
Lambertian  component  estimation  error  E,  can  be  limited  within  40%  if  we  remain  the 
parameter  errors  within  ±50%.  For  ±20%  errors,  we  can  keep  E,  within  10%.  From  the 
plots  given  in  Fig.  3.15,  we  also  found  that  the  averaged  specular  component  estimation 
error  Es  can  be  limited  within  approximately  100%  if  we  keep  the  parameter  errors  within 
±50%.  For  ±20%  errors,  we  can  keep  Es  within  40%.  Es  presented  in  Fig.  3.15  shows  a 
little  bit  excessive  amount  of  error  compared  to  the  previous  two  cases.  This  excessive 
error  is  originated  from  the  summation  in  (3.26.c)  over  the  region  where  fM  becomes  very 
low  intensity.  For  this  region,  a very  small  amount  of  estimation  error  can  make  a 
significant  contribution  to  the  entire  summation  results  because  the  absolute  error  is 
divided  by  fsi  which  is  very  small  in  this  region.  From  a practical  point  of  view,  these 
estimation  errors  cannot  be  an  obstacle  to  the  use  of  the  method  presented  in  this 
dissertation  in  practice.  To  assist  this  assertion,  we  have  already  discussed  the 
experimental  results  with  real  specular  objects  including  spherical  and  non-spherical 
samples.  Moreover,  the  following  simulation  results  on  surface  reconstruction  under 
parameter  estimation  errors  provide  more  visible  sense  for  that  assertion. 

In  Fig.  3.16,  we  illustrate  the  surface  reconstruction  results  under  the  parameter 
estimation  errors  only  for  the  case  of  yo=0.5  and  m=0.5.  As  expected  from  the  simulation 
results  on  Ez,  the  reconstructed  surface  Z(x,y)  shows  quite  accurately  the  shape  of  a 
hemisphere  for  20%  parameter  errors.  For  50%  parameter  errors,  a distorted  hemisphere 
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Fig.  3.13.  Error  analysis  results  for  Et.  Upper  part  denotes  3-D  plot  of  the  error 
and  the  lower  part  denotes  its  contour  map.  (a)  yo=0.3,  m= 0.3.  (b)  yo=0.5, 
m0= 0.5.  (c)  yo=0.7,  m= 0.7. 
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Fig.  3.13.  - Continued. 
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Fig.  3.13.  - Continued. 
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Fig.  3.14.  Error  analysis  results  for  E,.  Upper  part  denotes  3-D  plot  of  the  error 
and  the  lower  part  denotes  its  contour  map.  (a)  yo=0.3,  mo=0.3.  (b)  yo=0.5, 
ma= 0.5.  (c)  y=0.7,  m= 0.7. 
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Fig.  3.14.  - Continued. 
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Fig.  3.14.  - Continued. 
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Fig.  3.15.  Error  analysis  results  for  Es.  Upper  part  denotes  3-D  plot  of  the  error 
and  the  lower  part  denotes  its  contour  map.  (a)  yo=0.3,  m= 0.3.  (b)  yo=0.5, 
m „=0-5.  (c)  yo=0J,  m=0.1. 
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Fig.  3.15.  - Continued. 


87 


Es(%) 


Es(%) 


(C) 


Fig.  3.15.  - Continued. 
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Fig.  3.16.  Surface  reconstruction  results  under  parameter  errors  (yo=0.5,  m„=0.5).  (a) 
Ay/yo=20%,  Am/m  = 20%.  (b)  Ay/yo=50%,  Am/m  =50%. 
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which  resembles  the  shape  of  a square  has  been  obtained.  This  distorted  hemisphere  can 
still  be  recognized  as  a hemisphere  if  specular  images  are  properly  used  as  supplementary 
information  for  recognition.  These  experiments  have  been  performed  for  those  three  cases 
of  object  specularity  - (1)  highly  specular  (y=0.3,  m=  0.3),  (2)  medium  specular  (yo=0.5, 
m=  0,5),  and  (3)  near  Lambertian  (yo=0.7,  m=  0.7).  For  all  three  cases,  similar  results 
have  been  obtained. 


3.5  Summary 

In  this  chapter,  we  discussed  the  normal  vector  equalization  method  for  highlight 
separation  and  3-D  surface  description.  This  method  is  based  on  the  simplified  Torrance- 
Sparrow  model  for  specular  reflection  and  the  empirical  model  for  Lambertian  reflection. 
It  utilizes  a set  of  three  monochromatic  images  taken  from  three  different  locations  of 
illumination  sources,  separates  probable  highlights  from  the  images,  and  generates  a set 
of  three  Lambertian  images  and  a set  of  three  specular  images.  Surface  orientations  are 
obtained  from  the  Lambertian  images  by  solving  the  Lambertian  reflection  model  for 
surface  normal  vectors.  Surface  orientations  are  in  turn  provided  to  the  depth  recovery 
process  in  which  3-D  shape  of  a given  specular  object  is  derived.  The  problem  of 
practical  implementation  of  the  highlight  separation  algorithm  is  also  discussed  in  detail. 
Using  this  approach,  we  have  performed  computer  simulation  with  synthesized  images 
and  also  performed  experiments  with  real  objects.  Specimens  made  of  various  materials 
are  chosen  to  demonstrate  the  system  capability  for  a wide  range  of  materials  used  in  real 
applications.  From  the  results  obtained  from  simulation  and  experiments,  we  have 
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determined  the  following  facts:  (1)  The  proposed  approach  can  solve  the  highlight 
separation  problem  for  objects  which  are  made  from  various  materials  including  plastic 
and  metal.  (2)  It  can  also  be  used  to  obtain  3-dimensional  shape  description  such  as 
needle  map  and/or  3-D  depth  map  from  the  Lambertian  images.  (3)  Reflectance 
parameters  for  a given  specular  surface  can  be  measured  using  a sample  of  objects  or 
estimated  roughly  and  used  to  obtain  practically  useful  results  when  some  errors  are 
permitted  in  recovered  surface  shape,  Lambertian  components,  and  specular  components. 


CHAPTER  4 

MEASUREMENTS  OF  REFLECTANCE  PARAMETERS 
4.1  Introduction 

Extracting  3-D  surface  information  from  an  image  (or  images)  in  machine  vision 
[Hor75,  Woo78,  Ike81,  Par90b]  or  rendering  realistic  images  in  computer  graphics 
[C008I,  Bli77,  Pho75]  or  inspecting  surface  integrity  in  automated  optical  inspection 
[Chi88,  Par90a,  Par90c,  San88]  often  requires  a model  of  light  reflection  which  describes 
how  the  object  reflects  impinging  light  and  records  a pixel  at  the  corresponding  location 
of  the  image.  Up  to  date,  a number  of  light  reflection  models  have  been  proposed  for 
these  areas.  Some  of  them  include:  (1)  the  empirical  Lambertian  model  which  is  based 
on  the  cosine  law  [Hor86],  (2)  Phong’s  model  for  specular  reflection  [Pho75],  (3) 
Torrance-Sparrow’s  specular  reflection  model  which  is  based  on  geometrical  optics 
[Tor67],  (4)  the  dichromatic  reflection  model  of  Klinker  et  al.  which  captures  the  color 
difference  between  Lambertian  reflection  and  specular  reflection  [Kli87,  Kli88],  and  (5) 
Healey’s  approximate  color  reflection  model  [Hea88b]. 

Of  these  models,  we  adopted  the  empirical  Lambertian  reflection  model  and  the 
simplified  version  of  the  Torrance-Sparrow  model  throughout  this  research.  The  simplified 
Torrance-Sparrow  model  for  specular  reflection  is  known  as  a good  compromise  between 
the  accuracy  of  the  original  model  and  the  simplicity  of  the  algorithm  which  utilizes  the 
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model.  Some  successful  research  results  based  on  these  models  verify  their  usefulness  in 
practical  applications  such  as  highlight  separation  [Par90b],  inspection  [Par90c],  and 
computer  graphics  [C008I].  However,  one  obstacle  to  the  use  of  these  models  is  that  the 
reflectance  parameters  which  characterize  the  models  must  be  known  a priori. 

In  this  chapter,  we  will  address  the  problem  of  measuring  the  reflectance 
parameters  using  a sample  of  well-defined  shape  such  as  sphere  or  cylinder  which  has  the 
same  optical  characteristics  as  the  object  under  applications.  The  outcome  of  this  work 
can  be  used  to  obtain  3-D  surface  orientations  by  solving  the  modelling  equations  or  to 
draw  realistic  images  of  a simulated  surface,  as  we  will  discuss  later. 

It  is  relatively  simple  to  measure  the  reflectance  parameters  for  pure  Lambertian 
surfaces  or  for  completely  specular  surfaces.  For  the  Lambertian  surfaces,  reflectance  R 
in  (2.6)  can  be  measured  by  finding  peak  intensity  of  an  image  acquired  from  a spherical 
surfaces.  If  the  Lambertian  surface  is  heterogeneous,  i.e.,  made  from  different  materials 
for  whole  visible  area  of  the  surface,  R is  no  longer  a constant;  rather  it  becomes  a 
function  of  locations.  If  at  least  three  images  taken  from  three  different  locations  of  light 
sources  are  available,  an  analytical  method  can  be  used  to  calculate  the  reflectance  R for 
every  pixel  location  [Col82].  For  the  specular  surfaces,  the  reflectance  parameters  can  also 
be  obtained  by  measuring  peak  intensity  and  applying  a simple  fitting  method  [Wes87]. 

However,  for  a general  surface  which  contains  those  two  components  of  reflection 
simultaneously,  it  is  not  so  simple  to  determine  the  parameters.  Westphal  and  Nagel 
[Wes87]  attempted  to  fit  the  images  obtained  from  a cylinder  covered  with  white  paper 
to  a reflectance  model  derived  by  combining  the  empirical  Lambertian  reflection  model 
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with  Blinn’s  specular  model  [Bli77].  Nayar  and  Dceuchi  [Nay89]  used  photometric 
sampling  method  to  determine  3-D  shape  and  reflectance  properties  of  a general  surface 
ranging  from  perfectly  Lambertian  to  perfectly  specular. 

The  approach  discussed  in  this  chapter  utilizes  the  strict  dependence  of  the 
specular  reflection  component  upon  the  imaging  geometry  to  measure  the  reflectance 
parameters  for  a general  homogeneous  surface.  First  of  all,  it  measures  R'  which 
represents  a reflectance  constant  of  Lambertian  reflection  for  the  general  surface  in  which 
those  two  reflection  modes  are  present.  After  that,  K*  which  represents  peak  intensity  of 
specular  reflection  mode  for  the  general  surface,  is  measured  using  the  light  reflection 
model  employed  in  this  research.  As  we  discussed  in  Chapter  3,  R * and  K*  are 
mathematically  defined  as  R'=yR  and  K*=(l-y)K,  respectively,  based  on  the  relative 
weighting  factor  y,  reflectance  R for  pure  Lambertian  surface,  and  peak  intensity  K for 
pure  specular  surface.  The  parameter  m which  represents  surface  roughness  can  be 
measured  using  a simple  fitting  method  and  the  values  of  R * and  K*  measured  before.  The 
parameter  y which  represents  the  relative  weighting  factor  between  Lambertian  and 
specular  reflection  modes  can  be  measured  using  a sample  of  perfectly  Lambertian  surface 
or  perfectly  specular  surface.  Rough  estimation  of  y can  also  be  obtained  using  the  value 
of  R*  and  /t*. 

4.2  Reflectance  Parameter  Measurements 

Either  computer  vision  applications  or  computer  graphics  applications  which  are 
based  on  the  light  reflection  model  presented  in  (2.12)  require  to  determine  the  set  of 
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reflectance  parameters,  either  { R* ,K* ,m } or  [R,K,y,m\.  These  parameters  represent  optical 
characteristics  of  a surface.  For  a homogeneous  surface  which  is  made  from  a single 
material  for  whole  surface,  the  parameters  are  regarded  as  having  single  constant  values 
for  overall  surface  area.  In  this  section,  we  will  introduce  a newly  developed  method  for 
measuring  the  surface  parameters  of  a homogeneous  surface  using  a sample  of  a well- 
defined  shape  like  a sphere  [Par90d].  The  sample  specimen  must  be  made  from  the  same 
material  as  the  object  under  application  so  that  it  must  show  the  same  optical  character- 
istics as  the  object  under  application.  This  requirement  for  the  specimen  is  similar  to  that 
in  the  shape-from-shading  problems  in  which  a sphere  having  reflectance  properties 
identical  to  the  object  under  application  is  used  to  obtain  reflectance  maps  [W008O]. 

In  Fig.  4.1,  we  illustrate  the  whole  procedure  of  the  parameter  measurements.  They 
can  be  largely  grouped  into  three  stages.  The  first  stage  is  to  calibrate  the  intensity,  the 
azimuthal  angles,  and  the  elevation  angles  of  the  light  sources  so  that  the  peak  intensity 
measured  in  each  image  is  not  saturated  at  all  and  their  locations  are  widely  separated. 
The  second  stage  is  to  estimate  source  vectors  using  peak  intensity  locations  detected 
from  the  images.  The  third  stage  is  to  estimate  the  reflectance  parameters  based  on  the 
information  extracted  from  the  preceding  stage.  In  the  third  stage,  R * is  estimated  first. 
Next,  K*  is  estimated  using  the  peak  intensities  and  the  value  of  R'  estimated  just  before. 
The  surface  roughness  factor,  m is,  in  turn,  estimated  from  the  images  using  the  measured 
values  of  /?*  and  K* . 

We  will  first  discuss  the  image  acquisition  issues  and  then  discuss  how  to  measure 
the  surface  parameters  and  the  source  vectors.  Although  the  processes  of  source  vector 
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Fig.  4.1.  Processing  stages  for  reflectance  parameter  measurements. 
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measurements  are  performed  before  those  of  reflectance  parameter  measurements,  we  will 
discuss  the  former  processes  after  discussing  the  latter  processes. 

4.2.1  Image  Acquisition 

The  surface  parameter  measurement  technique  explained  in  this  dissertation  is 
based  on  three  monochromatic  images  obtained  from  three  different  locations  of  light 
sources.  In  Fig.  3.1,  we  illustrated  the  imaging  coordinate  system  including  light  sources 
and  a camera.  Three  azimuthal  angles  (9.)  - 0°,  120°,  and  240°  - are  chosen  for  each 
source  direction  S„  i=l,2,3.  For  elevation  angle  (0e),  approximately  55°  or  less  are  chosen 
for  all  three  sources.  The  ith  image  Ij(x,y)  is  obtained  in  association  with  the  ith  source 
direction  Sr  The  camera  and  light  sources  are  installed  as  far  from  the  specimen  as 
possible,  as  long  as  the  sensitivity  of  the  camera  allows,  to  satisfy  the  assumption  of 
orthographic  projection. 

Under  these  imaging  conditions,  we  can  ensure  that  one  highlight  associated  with 
one  of  the  light  sources  can  be  observed  in  the  corresponding  image,  because  we  use  a 
spherical  object  as  a specimen.  The  location  of  the  peak  intensity  in  each  highlight  area 
can  easily  be  found  using  a simple  algorithm  for  searching  maximum  intensity.  In  Fig. 
4.2,  we  denote  the  peak  intensity  locations  detected  in  the  ith  image  as  where 

i=  1,2,3.  Note  that  the  azimuthal  angles  of  the  detected  peak  locations  are  almost  same  as 
those  for  the  light  sources  used  to  take  the  images.  However,  due  to  the  inaccuracy  of  the 
device  holding  the  camera  and  the  light  sources,  the  measured  azimuthal  angles  may  be 
slightly  different  from  the  originally  set  values  - 0°,  120°,  240°.  Therefore,  a software 
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sphere  image 


highlight  area 

(0 


(a) 


(b) 


(c) 


Fig.  4.2.  Peak  intensity  locations;  in  order  to  ensure  that  these  peak  intensity  locations  are 
far  enough  apart  from  each  other,  elevation  angle  (9e)  of  55°  or  lower  is  recommended, 
(a)  I,  associated  to  source  S,.  (b)  l2  associated  to  S3.  (c)  I3  associated  to  Ss. 


98 


routine  for  measuring  the  azimuthal  angles  and  the  elevation  angles  for  each  source  is 
employed  in  the  first  part  of  the  entire  parameter  measurement  program.  In  addition, 
calibration  processes  are  carried  out  to  adjust  the  intensity  of  each  light  source  before 
starting  the  processes  of  the  parameter  measurements. 

4,2.2  Measurements  of  /?*  and  fC 

The  factors  which  affect  the  values  of  R * and  K*  include  (1)  surface  reflectance 
of  the  sample  object,  (2)  intensity  of  the  light  sources,  and  (3)  camera  sensitivity.  Of  these 
factors,  surface  reflectance  and  camera  sensitivity  are  equally  applied  to  all  three  images 
because  the  sample  has  a homogeneous  surface  and  all  the  images  are  acquired  by  one 
camera.  However,  the  second  factor  - light  intensity  - is  dependent  upon  the  source  and 
because  of  this,  R and  K*  in  (2.12)  are  also  dependent  upon  the  light  sources.  Therefore, 
we  will  now  begin  to  use  the  symbols  /?*,  and  It*,  in  association  with  the  ith  light  source. 
Namely,  from  (2.12)  the  ith  image  I,(x,y)  is  described  as: 

It(x,y)  = R*  St-N(x,y)+Kf  exp[-^^]  , *=1,2,3  (41) 

m2 

To  determine  R ; using  the  ith  image  I{x,y),  we  will  use  the  following  observation. 
Because  of  the  great  dependence  of  the  highlights  on  imaging  geometry,  I/xpi,ypi)  which 
means  the  image  intensity  observed  at  (xpi,ypi)  in  the  jth  image,  where  i*j,  mainly  consists 
of  the  Lambertian  component  of  reflection  provided  that  the  peak  intensity  locations  are 
far  enough  from  each  other  [Wes87,  Nay89],  Therefore,  from  (2.5),  (2.6),  and  (4.1), 
VW  becomes  as  follows  by  ignoring  the  specular  component: 
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7/vV  = yVvV+(1"y)VvV 

- ytyXpiJpi)  (4-2) 

= R]N{xpi,ylHySj  , i * j 

Meanwhile,  surface  normal  at  (xpi,ypi)  can  be  obtained  using  the  relation  Ot,(jr  -.y  -)=0  in 
(4.1),  because  the  intensity  at  (xpi,ypi)  is  at  a maximum  and  the  Lambertian  component  at 
( xpi,ypi ) can  be  neglected  in  this  case  unless  the  sample  is  highly  Lambertian.  That  is, 

«/VV  - m - 0 <4-3> 

and 


= Hi  = 


S,+  V 

Ivvf 


From  (4.2)  and  (4.4),  we  have: 


(4.4) 


7/VV  = Rj  HiSJ  ’ 1 * j 


(4.5) 


R* j can  be  found  from  (4.5): 


R'  - 7/W 

1 H,Sj 


* * J 


(4.6) 


where  H,  and  S,  are  provided  from  the  source  vector  measurement  stage.  Using  (4.6),  R" J 
is  obtained  from  the  ith  peak  intensity  location  in  the  jth  image.  A global  R'  for  all 
images  is  determined  as  follows  by  averaging  the  R*/s  obtained  from  all  combinations  of 
i and;',  where  / 2, 3 and  i*j: 
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= 1^  7/Wfi 


6£ 


1 >i 


<r 


fl  for  i*j 
[0  for  i-j 


(4.7) 


In  order  to  develop  a method  for  estimating  iC , we  use  another  observation  at  peak 
intensity  locations.  Hxpi,ypi),  which  means  the  intensity  of  the  ith  image  at  (xpi)y/M),  consists 
of  two  reflection  components  as  discussed  earlier  and  it  is  exactly  given  by: 


7<( vV  = 

* * a!(x  .,vj 


(4.8) 


Combining  (4.3)  and  (4.4)  with  (4.8),  we  have: 


From  (4.9),  £*,  is  determined  as: 


(4.9) 


K>  - s-  <410) 

where  /?*,  can  be  determined  using  (4.6).  A global  K*  for  all  images  can  be  found  by 

averaging  the  K* ' s obtained  from  the  ith  image  using  (4.10): 


(4.11) 


The  procedures  for  measuring  surface  parameters,  R*  and  AT*  can  be  informally 


described  as  follows: 


101 


Algorithm  2:  Reflectance  Parameter  Measurements  - 1. 

1.  Input:  images  /„  source  vectors  S„  bisectors  H„  peak  intensities  I{xpi,ypi),  and  peak 

intensity  locations  (xpi,yp;)  for  all  *=1,2,3. 

2.  Determine:  R*  and  R’  using  (4.6)  and  (4.7). 

3.  Determine:  K * and  K"  using  (4.10),  (4.11)  and  R*  determined  above. 

The  input  parameters  of  the  above  algorithm  are  provided  from  the  source  vector 
measurement  stage  which  will  be  discussed  later. 

In  order  to  measure  only  R * and  K*,  it  is  not  necessary  to  employ  a well-defined 
shape  such  as  sphere  or  cylinder  and  to  use  three  images  taken  from  three  different 
locations  of  light  sources.  As  long  as  the  highlights  measured  in  each  image  are 
geometrically  separated  far  enough  from  each  other,  it  is  possible  to  measure  R * and  K* . 
Minimum  requirements  for  that  include  (1)  the  surface  of  the  sample  must  be  curved  and 
smooth  and  (2)  at  least  two  images  taken  from  two  different  locations  of  light  sources  are 
needed.  We  carried  out  various  experiments  with  non-spherical  samples  to  measure  only 
R * and  K*. 

4.2.3  Measurements  of  m 

The  surface  roughness  is  characterized  by  m [Col81,  Tor67,  Hea88a].  In  order  to 
estimate  m for  a given  surface,  we  need  to  derive  a mathematical  representation  for 
surface  normal  vectors  of  a specimen.  If  a spherical  sample  is  employed,  its  surface 
normal  is  given  by: 
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NJx,y ) = (-A-) 


r r r 


(4.12) 


where  r denotes  the  radius  of  the  sphere  and  z,  which  is  given  by  z=('r2-x2-y2//2 » denotes 
the  surface  depth  of  the  sphere.  These  quantities  can  be  expressed  in  terms  of  either 
metric  unit  or  pixel  unit.  In  this  work,  we  use  the  pixel  unit,  because  we  can  easily  find 
the  radius  of  sphere  and  peak  intensity  locations  in  the  image  plane  which  is  specified  in 
terms  of  the  pixel  unit. 

As  given  in  the  previous  discussions,  the  ith  image  of  the  sphere  generated  from 
the  imaging  model  employed  in  this  work  is  given  by: 


l](x,y)  = % NJtoyySfKj  exp( 


-[cos 


) 


(4.13) 


m, 


where  //JC,y)  denotes  the  ith  modeled  image.  The  surface  roughness  factor  m;  is 
determined  for  the  ith  image  in  such  a way  that  it  minimizes  the  following  mean  absolute 
error  between  observed  image  J,{x,y)  and  modeled  image  Ij(x,y): 


F = J_  Y,  \I,(x,y)-f;(x,y)\  (4.14) 

M h xjrehigklight 

where  Mh  denotes  the  number  of  pixels  within  the  highlight  area.  A global  surface 
roughness  factor  m for  all  images  is  found  by  averaging  the  m,’s: 


m 


1 A 

= tE"1* 

j i=i 


(4.15) 


In  (4.14),  the  summation  is  carried  out  for  the  area  identified  as  highlight.  The 
highlight  area  can  be  estimated  using  the  peak  intensity  locations  (xpi,ypi)  as  follows:  The 
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highlight  area  for  the  ith  image  is  determined  as  a inner  area  of  a circle  whose  center  is 
given  by  (. x -,y  •)  and  its  radius  is  given  by  A rh,  as  illustrated  in  Fig.  4.3.  Note  that  this 
highlight  area  is  determined  in  such  a way  that  most  area  of  observed  highlight  should 
be  included  within  the  circle  while  it  is  confined  within  the  entire  sphere  image. 

The  procedures  for  measuring  the  surface  roughness  factor  m can  be  informally 
described  as  follows: 

Algorithm  3:  Reflectance  Parameter  Measurements  - II 

1.  Input:  /„  S„  H„  I{xpi,ypi),  (xpi,ypi),  R*,  K* , and  A rh  for  all  *'=1,2,3- 

2.  Determine:  mmm  and  mmax  which  bound  a possible  value  of  m such  that  mmin<m<mmax. 

3.  Do  while  (rnmin<m<mmax)  for  all  *=1,2,3. 

3.1  determine  highlight  area  using  (xp,,ypl)  and  A rh. 

3.2  determine  E:  using  (4.14)  for  the  highlight  area  and  store  £,  and  the  value  of  m,. 

4.  Find:  the  minimum  value  of  Ei  and  corresponding  value  of  m,. 

5.  Find:  the  average  of  m’s  using  (4.15). 

4.2.4  Measurements  of  v 

Although  the  surface  parameters  obtained  from  (4.7),  (4.1 1),  and  (4.15)  are  enough 
to  completely  characterize  the  reflection  model  given  by  (2.12),  it  is  often  interesting  to 
find  the  parameter,  y,  which  denotes  the  relative  strength  between  two  components  of 
reflection.  In  many  of  computer  graphics  literature,  it  is  used  to  classify  specular  objects 
based  on  the  degree  of  specularity  [Coo81]. 
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Fig.  4.3.  Determination  of  highlight  area  in  the  ith  image;  the  summation  in  (4.14)  is 
carried  out  for  the  highlight  area. 
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If  we  change  the  specimen  used  to  find  R\  K*,  and  m with  the  one  that  have 
almost  perfectly  specular  surface  in  the  same  imaging  conditions,  then  I,{xpi,ypi)  becomes 
to  contain  only  the  specular  reflection  component.  In  other  words,  the  new  specimen  has 
an  almost  perfectly  specular  surface  and  therefore  the  surface  parameter  y is 
approximately  equal  to  zero  in  this  case.  As  a result,  the  ith  image  intensity  for  the  new 
specimen  at  the  ith  peak  intensity  location,  denoted  as  7/jc  -,y  1^,  is  approximately 
given  by: 

VWU  ■ K <416> 

From  the  definition  of  A'*,  y ^ which  is  derived  from  the  ith  image  using  the  almost 
perfectly  specular  surface  is  given  by: 


OK 


-i.£ 

K 


Combining  (4.16)  with  (4.17),  we  have: 


(4.17) 


y.|,  = 1 ^ (4.18) 

iftppyp)  I Y"0 

where  K*,  can  be  determined  using  (4.10).  A global  y^  for  all  images  is  determined  by 
averaging  the  y|*’s  obtained  from  the  ith  image  using  (4.18): 

Another  way  to  determine  y is  to  exchange  the  specimen  used  to  measure  R\  fC, 
and  m with  the  one  that  has  almost  perfectly  Lambertian  surface  rather  than  perfectly 
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specular  surface.  In  this  case,  y becomes  almost  one  and  therefore  I,{xpi,ypi)  1^.,  is 
approximately  given  by: 

VWI,-.  - R (4-20) 

From  the  definition  of  R\  y |*,  which  is  derived  from  the  ith  image  using  Lambertian 
surface,  is  given  by: 


I * 

Y,U-  R 

Combining  (4.20)  with  (4.21),  we  have: 


(4.21) 


i y — 1 


(4.22) 


where  /?*,  can  be  determined  using  (4.5).  A global  y for  all  images  is  determined  by 
averaging  the  y|*’s  obtained  from  (4.22): 


vl«={Ev,l»  (4'23) 

The  procedures  for  measuring  the  relative  weighting  factor,  y can  be  informally 
described  as  follows: 

Algorithm  4:  Reflectance  Parameter  Measurements  - PI 

1 . Change:  the  specimen  with  the  one  that  has  almost  perfectly  specular  surface  or  almost 

perfectly  Lambertian  surface  and  take  three  images  from  either  one. 

2.  Find:  peak  intensities  or  I,<xpi,ypi)  1^,. 

3.  Find:  y|*  using  (4.18)  and  (4.19)  or  y|*  using  (4.22)  and  (4.23). 
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4.2.5  Estimation  of  Source  Vectors 

The  source  vectors  Sit  i=  1,2,3,  can  be  obtained  using  (4.4)  and  (4.12).  From  (4.12), 
surface  normal  vectors  at  the  peak  intensity  locations  are  given  by: 


jy vV = 


r~2  2 2" 

**  ^ y -xpi-ypi 


r r 


Combining  (4.4)  with  (4.24),  we  have  for  H,: 


«i- 




9 

r r 


n 2 

ypt  v -xPi-ypi 


(4.24) 


(4.25) 


5,  can  be  found  from  H,  using  the  relation  given  in  (4.4).  Since  V=(0,0,1),  we  have  from 

(4.4): 


(■vyi*1^) 


- 


Since  Six  +Siy2+Siz2=  1,  we  have: 


(SjjSj,  1 +lS  . ) 

■ ( w) 


(4.26) 


(4.27) 


After  performing  some  mathematical  manipulation,  we  have  the  following  equations  for 
source  vector  S,: 
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\ = F™*H* 


(4.28. a) 


5<y  = 


(4.28.b) 


Su  = 2Hl-\ 


(4.28.C) 


Based  on  the  above  discussions,  the  procedures  for  measuring  source  vectors  can 
be  informally  described  as  follows: 

Algorithm  5:  Source  Vector  Measurements 

1.  Input:  acquired  images  /,  for  all  i=l,2,3  and  the  radius  of  spherical  sample,  r. 

2.  Run:  a subprogram,  PEAKSEARCH  for  all  images  to  find  peak  intensity  locations, 

(x^)  for  all  i'=l, 2,3. 

3.  Determine:  H,  using  (4.25)  for  all  i=  1,2,3. 

4.  Determine:  S,  using  (4.28)  for  all  /=  1,2,3. 


In  this  chapter,  we  have  selected  the  same  samples  as  those  used  in  Chapter  3 to 
conduct  experiments  for  measuring  reflectance  parameters.  We  also  used  the  same  image 
processing  system  illustrated  in  Fig.  3.4. 

As  in  the  case  of  Chapter  3,  we  also  need  to  calibrate  the  sensitivity  of  the  camera 
and  the  locations  and  the  intensity  of  the  light  sources  so  that  the  resultant  images  make 
use  of  the  full  available  dynamic  range  of  the  image  processing  system  and  the  locations 
of  the  peak  intensity  measured  in  the  images  are  all  separated  far  enough  apart  from  each 
other.  In  Fig.  4.4,  we  illustrate  the  images  of  the  spherical  samples  taken  after  performing 


4.3  Experiments  with  Real  Samples 
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Fig.  4.4.  The  images  acquired  from  Sphere  A,  B,  and  C from  top  to  bottom.  The  images 
of  Sphere  A has  the  resolution  of  142X142  pixels  and  those  of  Sphere  B and  C have 
95X95  pixels. 
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the  calibration  processes.  The  images  of  Sphere  A have  the  size  of  142X142  pixels  and 
the  images  of  Sphere  B and  C have  that  of  95X95  pixels. 

In  order  to  measure  the  source  vectors,  we  used  Sphere  A because  it  shows  the 
most  prominent  specular  characteristics  among  the  spherical  samples.  In  Table  4.1,  we 
present  the  measured  source  vectors  set  to  measure  reflectance  parameters  of  the  spherical 
samples.  They  are  described  using  the  azimuthal  angles  (9.)  and  the  elevation  angles  (0e) 
defined  in  Fig.  3.1. 

Measurement  results  of  /?’  and  K*  for  Sphere  A,  B,  and  C are  also  summarized  in 
Table  4.1.  The  data  include  the  values  of  R*'s  and  K* ' s derived  from  the  ith  image  for 
all  i=l,2, 3.  As  we  discussed  earlier,  Sphere  A,  B,  and  C are  regarded  as  the 
representatives  of  highly  specular,  medium  specular,  and  near  Lambertian  classes  of 
surfaces,  respectively.  According  to  the  experimental  results  given  in  Table  4.1,  we  can 
classify  those  types  of  surfaces  using  the  values  of  R * and  X*.  That  is,  we  can  classify  the 
surface  as  highly  specular  if  7?*<<X*,  as  medium  specular  if  R*~K*,  as  near  Lambertian 
if  f?*>>X*.  These  observations  can  be  used  to  derive  a rough  estimation  of  y which  is  not 
based  on  the  images  of  perfectly  Lambertian  or  perfectly  specular  surfaces.  When  we  take 
an  image,  if  we  adjust  the  brightness  of  the  image  with  the  full  dynamic  range  of  the 
image  processing  system,  R and  K are  often  limited  by  the  maximum  brightness  of  the 
system.  That  is,  we  may  have  R-K-255,  if  the  image  processing  system  has  8-bit 
resolution.  Then,  from  the  definition  of  R * and  K*,  the  rough  estimation  of  y,  denoted  as 
y*  is  given  by: 
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Table  4.1.  Measurement  results  of  R*,  K* , and  source  vectors  for  the  spherical  samples. 


Sphere  A 

Sphere  B 

Sphere  C 

Peak 

Intensity 

Ii(xpi>ypi) 

254 

254 

187 

h(\2>yPi) 

254 

220 

167 

h(xP3>yP3) 

254 

232 

176 

Lambertian 

r; 

103.4 

132.4 

205.8 

Reflectance 

r; 

84.1 

114.6 

153.4 

r; 

89.6 

109.9 

175.2 

R* 

92.4 

119.0 

178.1 

Peak 

k; 

166.6 

141.3 

0 

Specular 

Reflection 

r; 

165.4 

105.3 

19.9 

k; 

166.6 

118.3 

10.3 

K* 

166.2 

121.6 

7.51 

Source 

Vectors 

(0.1.0..) 

2.49  , 

52.16 

(degree) 

(e^) 

122.91  , 

56.86 

(0a3,ee3) 

235.62  , 

52.16 

Table  4.2.  Measurement  results  of  /?*,  fC , and  source  vectors  for  the  non-spherical  samples. 
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Y = 


R* 


Rm+K* 


(4.29) 


Westphal  and  Nagel  [Wes87]  defined  the  relative  weighting  factor  in  such  a way  that 
R=K. 


The  measurement  results  of  R \ K*,  and  source  vectors  for  the  non-spherical 
samples  are  presented  in  Table  4.2.  In  measuring  /?*  and  K*  for  the  Transistor,  R*  and  AT/ 
are  abandoned  in  calculating  the  averages,  R * and  K*,  because  their  values  are  exceedingly 
different  from  the  other  two  values.  This  is  because  the  surface  of  the  Transistor  is  not 
highly  curved  so  that  the  highlight  observed  in  Ij  is  not  virtually  diminished  to  zero  at  the 
other  two  locations  of  the  highlights.  This  results  in  the  extraordinary  values  of  R*  and 

k;. 

In  order  to  measure  the  surface  roughness  m,  we  make  use  of  the  values  of  /?,*’  s 
and  K*’s  in  Table  4.1  to  determine  I,{x,y)  in  (4.13).  The  radius  of  the  highlight  area  is 
determined  as  9 (pixel)  for  Sphere  A and  5 (pixel)  for  Sphere  B.  For  Sphere  C,  we  arbi- 
trarily determined  the  radius  as  12  (pixel),  which  is  greater  than  that  of  Sphere  B,  because 
Sphere  C has  the  characteristics  of  near  Lambertian.  For  Sphere  C,  the  value  of  m is  not 
so  influential  to  the  entire  image  intensity  as  those  of  Sphere  A and  B,  because  there  is 
no  prominent  specularity  and  R'»lC  in  this  sample.  In  Table  4.3,  we  present  the 
measurement  results  of  tn  for  the  spherical  samples.  To  illustrate  how  accurately  the 
algorithms  approximate  the  measured  image  IJx.y)  with  the  modeled  image  I{x,y)  around 
the  highlight  area,  we  present  3-D  plots  of  I fay),  i,(x,y),  and  \II(x,y)-I1(x,y)\  for  Sphere 
A and  B in  Fig.  4.5.  Sphere  C is  excluded  here  because  it  shows  no  prominent  highlight. 


Mil 

(a) 


Ml 

(b) 


Fig.  4.5.  To  determine  m„  /,  is  fitted  to  /,  so  that  their  absolute  error  |/.-/.|  is  minimized. 
These  plots  are  drawn  with  the  same  scale  for  easy  comparison,  (a)  Sphere  A.  (b)  Sphere 
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Table  4.3.  Measurement  results  of  m (radian)  for  the  spherical  samples. 


Sphere  A 

Sphere  B 

Sphere  C 

mi 

0.031 

0.083 

0.53 

m2 

0.032 

0.081 

0.55 

m} 

0.035 

0.091 

0.58 

m 

0.032 

0.085 

0.55 
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Table  4.4.  Measurement  results  of  y for  Sphere  B. 


i=  1 

i=2 

i=3 

J,<Xpi>y pi)  \y-0 

254 

231 

237 

K- 

85.7 

75.0 

89.5 

Ylit 

0.66 

0.67 

0.62 

Y \k 

0.65 

HxPi,yPi)  1^, 

91 

83 

74 

K 

50.0 

42.5 

35.7 

YiU 

0.55 

0.51 

0.48 

YIj. 


0.51 
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Although  y is  not  an  essential  component  for  the  imaging  equation  given  in  (4.1), 
we  performed  a simple  experiment  to  obtain  a rough  estimate  of  y for  Sphere  B.  The 
reason  we  had  a rough  value  for  Sphere  B is  that  neither  a completely  specular  surface 
nor  a completely  Lambertian  surface  was  available  for  the  experiment.  As  a result,  we 
used  Sphere  A and  C as  substitutes  for  completely  specular  and  Lambertian  surfaces, 
respectively.  In  Table  4.4,  we  summarize  the  results  for  measuring  y for  Sphere  B. 
Although  the  experiment  was  not  conducted  under  precise  conditions,  we  can  estimate 
that  the  approximate  value  of  y for  Sphere  B is  around  0.55.  In  Table  4.5,  we  also  present 
the  value  y*  which  is  obtained  using  (4.29)  for  all  the  samples  employed  in  this 
experiment.  Especially,  the  similarity  between  y|K,  y|R,  and  y*  for  Sphere  B shows  that 
any  one  of  the  estimates  of  y can  be  used  to  categorize  the  underlying  sample  based  on 
the  degree  of  specularity. 

In  order  to  test  how  accurately  the  measured  reflectance  parameters  characterize 
the  actual  imaging  equation  given  in  (4.1),  we  performed  the  following  experiments:  The 
first  one  is  to  test  the  validity  of  the  parameters  by  computer  generated  images.  The  sets 
of  images  generated  by  using  the  measured  parameters  in  (4.1)  are  compared  with  the  real 
images  from  which  the  parameters  are  measured.  In  Fig.  4.6,  we  present  the  sets  of 
synthesized  images  and  the  real  images  for  Sphere  A,  B,  and  C.  The  similarity  between 
the  images  proves  that  the  measured  parameters  and  the  imaging  model  given  in  (4.1) 
reflect  the  real  situation  quite  accurately. 
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Table  4.5.  Measurement  results  of  y*  using  (4.29)  for  all  samples. 


Spherical  Sphere  A Sphere  B Sphere  C 

Samples 

0.36  0.49  0.96 


Non-spherical  Crayon  Transistor  Button  Bulb  Capsule 

Samples 


0.46 


0.34 


0.41 


0.61 


0.55 
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(a) 


Fig.  4.6.  Real  images  and  synthesized  images.  The  images  in  the  left  column  denote  the 
real  images  /,  and  the  images  in  the  right  column  denote  the  synthesized  images  /„  for 
*=1,2,3  from  top  to  bottom,  (a)  Sphere  A.  (b)  Sphere  B.  (c)  Sphere  C. 
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The  second  test  is  to  use  the  measured  parameters  for  the  highlight  separation 
techniques  which  is  based  on  the  normal  vector  equalization  method  [Par90b],  as 
discussed  in  the  experimental  results  given  in  Chapter  3. 

The  third  test  is  to  determine  /?*  using  only  the  Lambertian  reflection  model  and 
compare  it  with  the  R ’ presented  in  Table  4.1  and  4.2.  Unless  the  sample  is  made  up  of 
planes  and  one  of  the  planes  is  oriented  in  such  a way  that  highlight  occurs  for  one  part 
of  the  entire  plane  area,  the  intensity  of  most  of  the  image  area  consists  of  Lambertian 
component  except  for  the  bright,  small  highlight  area.  The  samples  used  in  this 
experiment  satisfy  the  above  condition.  Then,  assuming  a Lambertian  surface,  R * can  be 
found  using  the  constraint  of  unit  normal  vector  as  follows  [Col82]. 

r*  = \srli\  (4-30> 

where  we  ignore  the  difference  between  the  intensity  of  the  ith  light  source  and  have 
/?,*=/?*  for  all  i=l,2,3.  To  determine  /?*  using  (4.30),  we  first  segment  the  area  of  the 
sample  from  the  background.  As  illustrated  in  the  highlight  separation  results  given  in 
Chapter  3,  the  backgrounds  are  characterized  by  extremely  dark  intensity.  We  employed 
a binarization  algorithm  developed  by  Otsu  [Ots78]  to  segment  the  sample  from  the 
background.  In  Fig.  4.7,  several  examples  of  the  segmentation  results  are  illustrated.  For 
the  area  identified  as  a sample,  (4.30)  is  applied  and  the  histogram  of  the  R * is  obtained. 
In  Fig.  4.8,  a couple  of  examples  of  the  histograms  are  presented.  Since  the  Lambertian 
reflection  occurs  for  most  area  of  the  image,  R * can  be  determined  as  the  one  for  which 
the  histogram  has  its  maximum  value.  In  Table  4.6,  we  present  the  value  of/?*  determined 
by  using  (4.30).  The  proximity  between  the  value  of  /?*  given  in  Table  4.1,  4.2,  and  4.6, 
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Fig.  4.7.  Examples  of  the  segmented  images.  The  white  area  denotes  the  sample  and  the 
black  area  denotes  background,  (a)  Spherical  samples,  (b)  Non-spherical  samples. 


Frequency  Frequency 


Sphere  A 


0 40  80  ISO  180  200  240 


R‘ 

(a) 


Transistor 


/?* 


(b) 

Fig.  4.8.  Examples  of  the  histograms,  (a)  Sphere  A.  (b)  Transistor. 
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Table  4.6.  Measurement  results  of  R*  using  (4.30).  Threshold  level  (Thr)  denotes  the 
intensity  level  used  to  segment  the  image. 


Spherical 

Samples 

Sphere  A 

Sphere  B 

Sphere  C 

Thr 

44 

61 

87 

R' 

89 

119 

190 

Non-spherical  Crayon  Transistor  Button  Bulb  Capsule 

Samples  ■ 

Thr  50  40  67  72  59 


102 


82 


103 


143 
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also  proves  that  the  proposed  method  can  provide  reflectance  parameters  reliable  and 
accurate  enough  for  other  real  applications  such  as  computer  graphics,  3-D  specular  object 
inspection,  and  highlight  separation. 

4.4  Error  Analysis  of  the  Algorithms 

The  starting  point  for  measuring  surface  parameters  is  to  find  R * from  which  other 
parameters  are  derived.  For  instance,  if  is  found  using  R*  and,  in  turn,  m is  found  using 
R*  and  if.  Therefore  the  measurement  error  of  R * can  propagate  into  all  the  parameters. 
Essentially  the  accuracy  of  R * is  affected  by  the  degree  of  the  specularity  of  the  sample, 
because  the  approximate  equation  in  (4.2)  assumes  that  the  highlight  observed  in  the 
sample  is  quite  sensitive  to  imaging  geometry.  In  other  words,  for  the  surfaces  which 
have  small  value  of  m and  y,  the  specular  component  of  reflection  rapidly  reduces  to  zero 
as  a deviates  from  zero.  As  a result,  the  specular  component  Is/xpi,ypi)  in  (4.2)  can  be 
negligible,  so  that  the  approximate  relation  holds  more  accurately.  Therefore,  for  more 
specular  surfaces,  more  accurate  measurement  of  /?*  can  be  obtained  using  the  algorithms. 
Of  course,  more  accurate  R * increases  the  accuracy  of  other  parameters  such  as  if,  m,  and 
y.  As  we  will  discuss  below,  some  simulation  results  for  the  parameter  estimation 
algorithms  verify  this  dependence  of  the  accuracy  of  the  algorithm  upon  the  value  of  m 
and  y. 

In  Fig.  4.9,  we  illustrate  the  simulation  processes  for  studying  the  inherent  error 
bound  of  the  algorithms  presented  in  this  dissertation.  The  set  of  original  reflectance 
parameters,  \y0,m0Jt0,K0\,  is  used  to  synthesize  the  images  which  will  be  processed  to 
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Fig.  4.9.  Simulation  processes  for  measurement  error  analysis. 
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estimate  the  original  surface  parameters.  We  will  denote  the  estimates  of  the  original 
surface  parameters  as  me,  /sT/,and  R*.  The  weighting  factor,  y,  is  not  estimated  in  this 
simulation  because  it  is  not  an  essential  factor  to  completely  characterize  the  reflection 
model  given  in  (4.1). 

The  relative  estimation  errors  defined  as  follows  have  been  obtained  for  (y,m) 
space  in  the  simulation  work.  The  results  are  presented  in  Fig.  4.10. 


where  ER.,  EK„  and  Em  denote  relative  estimation  errors  of  the  parameters  - /?*,  K*,  and 
m,  respectively.  In  (4.31),  R ’ and  K*  are  defined  as  R*=y„R0  and  K*=(l-yJK0, 
respectively. 

As  discussed  earlier,  the  approximate  equations  in  (4.2)  hold  more  accurately  for 
more  specular  objects.  As  a result  of  this,  the  plots  shown  in  Fig.  4.10  show  that  the 
parameter  estimation  errors  become  more  negligible  for  more  specular  objects.  From  these 
plots,  we  can  determine  that  the  parameter  measurement  algorithms  presented  in  this 
dissertation  can  estimate  the  reflectance  parameters  (1)  within  20%  relative  error  for 
most  categories  of  specular  objects,  (2)  1?  within  20%  for  highly  specular,  medium 


- 


\K-K  I 
K 


(4.31. a) 


(4.3  l.b) 


(4.3  l.c) 
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(a) 

Fig.  4.10.  Error  bounds  of  the  algorithm  for  reflectance  parameter  measurements. 
The  upper  part  denotes  3-D  plot  of  the  error  and  the  lower  part  denotes  its  contour 
map.  To  generate  images,  source  vectors  (0  =0°, 120°, 240°  and  0e=45°)  are  simulat- 
ed for  each  image,  (a)  £*..  (b)  EK„  (c)  Em. 
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Fig.  4.10.  - Continued. 


Fig.  4.10.  - Continued. 
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specular,  and  near  Lambertian  surfaces,  and  (3)  m within  10%  for  the  surfaces  as  in  the 
K*.  In  other  words,  the  algorithms  discussed  in  this  chapter  can  estimate  the  surface 
parameters  of  a specimen  with  at  most  20%  relative  error  unless  the  specimen  is  highly 
Lambertian,  in  which  yo>0.7  and  mo>0.1  [Par90d]. 

From  practical  point  of  view,  these  parameter  errors  cannot  be  an  obstacle  to  the 
use  of  the  parameter  measurement  algorithm  in  practice.  To  assist  this  assertion,  we  have 
already  discussed  the  validity  of  the  measured  parameters  in  such  application  fields  as 
computer  graphics  and  highlight  separation,  which  are  all  based  on  the  imaging  model 
given  in  (4.1). 


4.5  Summary 

In  this  chapter,  we  have  presented  a new  method  for  measuring  reflectance 
parameters  of  a homogeneous  surface.  These  parameters  include  (1)  /?*  which  represents 
Lambertian  reflectance,  (2)  fC  which  represents  peak  specular  reflection,  (3)  m which 
represents  surface  roughness,  and  (4)  optionally  y which  represents  the  relative  weighting 
factor  between  Lambertian  and  specular  reflection  modes.  The  method  starts  from  three 
images  taken  from  three  different  locations  of  light  sources  and  derives  the  reflectance 
parameters  automatically.  Based  on  the  strict  dependence  of  the  specular  reflection  upon 
the  imaging  geometry,  it  first  measures  R *,  which  represents  a reflectance  constant  of 
Lambertian  reflection.  After  that,  K*,  which  represents  peak  intensity  of  specular  reflection 
mode  for  the  general  surface,  is  measured  using  the  light  reflection  models.  The  parameter 
m,  which  represents  surface  roughness,  is  measured  using  a simple  fitting  method  and  the 
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values  of  /?*  and  fC  measured  before.  The  parameter  y,  which  represents  the  relative 
weighting  factor  between  Lambertian  and  specular  reflection  modes,  can  also  be  measured 
using  a sample  of  perfectly  Lambertian  surface  or  perfectly  specular  surface.  Rough 
estimation  of  y is  given  by  using  the  value  of  R*  and  K* . 

Although  we  conducted  experiments  using  a total  of  three  images,  actually  two  are 
the  minimum  number  of  images  which  must  be  supplied  to  the  method.  This  is  because 
we  need  at  least  two  images  to  find  R*  using  the  intensity  of  the  jth  images  at  the  ith 
peak  intensity  location,  where  fcj.  Once  we  find  R*,  a minimum  of  one  image  is  needed 
to  compute  other  parameters.  However,  in  this  work,  we  have  decided  to  use  three  images 
to  obtain  more  reliable  global  parameters,  which  are  obtained  by  averaging  the  individual 
parameters  estimated  from  each  image,  as  explained  in  (4.7),  (4.11),  and  (4.15).  Another 
merit  of  using  three  images  is  that  we  can  apply  the  images  to  the  highlight  separation 
algorithm  presented  in  Chapter  3 and  test  the  validity  of  the  measured  parameters. 

Experimental  results  obtained  from  several  real  spherical  and  non-spherical 
samples  show  that  this  method  is  practical  for  applications  to  many  other  fields. 
Simulation  results  on  parameter  estimation  errors  give  us  confidence  that  the  method 
produces  results  which  are  accurate  enough  for  practical  applications,  unless  the  sample 
is  highly  Lambertian.  For  a highly  Lambertian  surface,  it  is  very  simple  to  measure  the 
surface  parameters.  In  this  case,  we  only  measure  R using  (4.20)  and  have  K equal  zero. 

Several  possible  areas  for  further  applications  are  listed  as: 

(1)  Inspection:  Object  inspection  often  assumes  that  the  objects  under  inspection  are 
already  known  and  they  are  always  placed  in  stable  environments.  This  method 
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can  serve  as  an  important  step  in  object  inspection  by  providing  reflectance 
parameters  which  can  be  used  to  detect  defective  objects. 

(2)  Computer  vision:  The  main  problem  in  computer  vision  is  to  describe,  represent, 
and  recognize  real  objects.  For  the  computer  vision  algorithms  which  are  based 
on  the  physical  reflection  models,  this  method  can  serve  as  a useful  tool  to 
provide  the  reflectance  parameters  to  the  algorithms. 

(3)  Computer  graphics:  One  of  the  main  interests  in  computer  graphics  is  to  generate 
images  which  resemble  real  images  as  closely  as  possible.  For  computer  graphics 
applications  which  are  based  on  the  reflection  model,  it  is  also  necessary  to 
determine  the  reflectance  parameters  as  accurately  as  possible.  Our  method  can  be 
used  to  estimate  the  parameters  of  the  surface  which  is  portrayed  by  the  graphic 
algorithm. 


CHAPTER  5 

APPLICATION : AUTOMATIC  INSPECTION  OF  ELECTRONIC  COMPONENTS 

5.1  Introduction 

A printed  circuit  (PC)  board  is  a basic  building  block  of  many  electronic  devices. 
PC  boards  are  inspected  extensively  before  the  insertion  of  components  and  the  soldering 
process.  A great  deal  of  attention  has  been  spent  on  this  single  inspection  application  in 
the  past  ten  years  [Chi88].  Although  a great  deal  of  work  has  been  done  in  the  area  of 
bare  PC  board  inspection,  very  little  research  has  been  addressed  in  the  inspection  of 
assembled  PC  boards,  which  includes  that  of  solder  joints  and  electronic  components 
mounted  on  a PC  board.  This  is  primarily  due  to  the  difficulty  of  the  problem.  Whereas 
PC  patterns  are  essentially  2-dimensional,  solder  joints  and  electronic  components  have 
3-dimensional  structures.  Moreover,  their  specular  nature  combined  with  great  variability 
of  appearance  in  acquired  images  makes  successful  inspection  exceedingly  difficult 
[Tou88,  Par89,  Par90a,  Cap88,  Bar85,  Dav88,  Che88,  Kel88,  Cap90,  Bad88,  Uni90]. 

In  this  chapter,  we  will  present  the  design  of  an  automated  inspection  system  for 
electronic  components  assembled  on  a PC  board.  The  inspection  system  has  been 
developed  not  only  as  an  integral  part  of  the  PCB/AID  System  (Automated  Inspection  and 
Diagnosis  System  for  PC  Board  Manufacturing)  but  also  as  an  example  of  the  practical 
applications  of  the  basic  research  work  presented  in  the  previous  chapters.  The  highlight 
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separation  method  discussed  in  Chapter  3 and  the  method  for  measuring  reflectance 
parameters  discussed  in  Chapter  4 will  be  applied  to  the  problem  of  inspecting  electronic 
components.  Another  possible  application  to  specular  object  recognition  will  be  briefly 
discussed  in  Chapter  6 as  a suggested  research  for  future  work.  The  roles  of  the  PCB/AID 
System  are  (1)  automating  visual  inspection  processes  which  are  usually  performed  by 
human  inspectors  in  most  industries  and  (2)  utilizing  the  inspection  results  provided  from 
the  automated  inspection  systems  in  a fault  diagnosis  system  to  automatically  control  the 
manufacturing  processes.  The  PCB/AID  System  consists  of  a solder  joint  inspection 
system  [Par89,  Par90a],  an  electronic  component  inspection  system  [Par90c],  and  a fault 
diagnosis  system  [Cha84,  Tou83,  Chu91].  In  this  chapter,  we  will  restrict  our  attention 
to  the  problem  of  inspecting  electronic  components  assembled  on  a PC  board,  using  the 
basic  frameworks  given  in  Chapter  3 and  Chapter  4.  The  other  two  systems  will  be 
considered  in  the  Appendix. 

In  the  past  few  years,  several  approaches  for  inspecting  components  on  assembled 
PC  boards  have  been  announced.  Barnard  [Bar85]  developed  an  inspection  system  for 
determining  integrity  of  components  and  for  verifying  part  identification  markings  using 
grey-scale  and  color  images.  He  also  developed  a component  inspection  system  which 
makes  use  of  range  data  obtained  by  structured  light  techniques.  Davis  et  al.  [Dav88] 
have  developed  an  inspection  system  for  SMT  (Surface  Mount  Technology)  components. 

I 

Their  system  uses  infrared  lasers  as  a structured  light  source  to  detect  missing  or 
misplaced  components.  Chemaly  [Che88]  announced  a method  for  inspecting  surface 
defects  on  integrated  circuit  packages,  based  on  mathematical  morphology.  Kelley  et  al. 
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[Kel88]  at  Synthetic  Vision  Systems  have  developed  an  inspection  system  for  SMT 
components  using  3-D  triangulation  techniques.  Other  attempts  include  Capson  and 
Tsang’s  automated  visual  measurement  system  for  SMT  component  inspection  [Cap90], 
the  3-D  vision  system  for  component  lead  inspection  developed  by  Badami  et  al.  [Bad88], 
and  Universal  System’s  component  inspection  system  which  uses  multiple  LED  sources 
to  detect  missing,  misplaced,  and  not-soldered  components  [Uni90].  Performance  of  those 
academic  systems  are  not  completely  satisfactory  in  real  applications  and  the  commercial 
systems  are  quite  expensive  and  complex. 

In  this  chapter,  we  will  approach  the  problem  of  component  inspection  by 
considering  the  light  reflection  components,  which  generally  consist  of  the  Lambertian 
and  specular  components  of  reflection.  As  well  as  many  practical  tasks  in  industrial  vision 
problems,  inspecting  electronic  components  assembled  on  a PC  board  requires 
interpretation  of  the  images  of  specular  surface.  Many  shiny  components  on  PC  boards 
produce  specular  reflections  which,  in  turn,  generate  highlights  in  the  images.  As  we 
discussed  in  Chapter  1,  the  specular  reflection  has  two  opposing  properties:  (1)  the 
information  aspect  - it  can  be  used  to  determine  shape  of  objects  and  (2)  the  distortion 
aspect  - it  can  induce  numerous  problems  for  the  algorithms  which  are  based  on 
correspondence  of  images  (motion  and  stereo)  or  variation  of  image  intensity  (edge 
detection)  [Par90b,  Par90e].  We  have  designed  the  component  inspection  system  based 
on  a unique  approach  considering  those  two  aspects  of  highlights  separately.  We  first 
separate  the  highlights  from  acquired  raw  images  using  the  normal  vector  equalization 
method  discussed  in  Chapter  3 and  obtain  two  sets  of  highlight-only  (specular)  images 
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and  highlight-free  (Lambertian)  images.  In  the  dual  channel  processing  strategy,  the 
Lambertian  images,  in  which  the  distortion  aspect  is  completely  removed,  and  the 
specular  images,  in  which  the  information  aspect  is  completely  conserved,  are  separately 
processed  to  extract  and  utilize  as  much  information  concealed  in  those  two  sets  of 
images  as  possible.  Moreover,  the  method  for  measuring  reflectance  parameters,  which 
is  discussed  in  Chapter  4,  is  employed  in  this  system  to  obtain  the  reflectance  parameters 
for  each  component  on  a PC  board.  The  measured  reflectance  parameters  are  supplied  to 
the  normal  vector  equalization  method  as  a priori  information. 

In  Fig.  5.1,  we  illustrate  one  of  the  rationales  for  employing  those  unique 
approaches  to  separate  highlights  and  extract  the  features  for  inspecting  components.  For 
instance,  if  we  attempt  to  detect  a tilted  component  without  relying  on  any  kind  of  range 
sensors,  it  becomes  very  difficult  to  check  the  defect.  However,  we  can  overcome  this 
difficulty  by  using  the  features  extracted  from  the  specular  reflection  component.  As 
illustrated  in  Fig.  5.1(a),  a normal  component  reflects  most  of  the  impinging  light  by 
specular  reflection  mode,  resulting  in  bright  highlights  in  the  specular  images,  since  the 
light  source  is  positioned  almost  uprightly.  However,  for  a tilted  component  illustrated  in 
Fig.  5.1(b),  most  of  the  specularly-reflected  light  does  not  contribute  to  the  image 
intensity,  due  to  the  great  dependence  of  the  specular  reflection  mode  upon  the  imaging 
geometry.  As  a result,  the  brightness  of  the  specular  image  of  a tilted  component  is  quite 
different  from  that  of  a normal  component.  These  differences  can  be  used  to  detect  and 
categorize  the  tilted  component. 
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(b) 

Fig.  5.1.  Behavior  of  highlight  in  (a)  a normal  component  and  (b)  a tilted  component. 
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Whereas  the  specular  reflection  components  provide  us  with  quite  sensitive 
information  regarding  the  mounting  status  of  the  component  under  inspection,  Lambertian 
components  provide  quite  reliable  information  regarding  existence  of  the  component, 
because  the  Lambertian  component  is  not  greatly  affected  by  imaging  conditions  such  as 
light  source  intensity,  surface  orientation  of  the  component,  and  imaging  geometry.  For 
instance,  the  specular  images  presented  in  Fig.  5.1  show  a big  difference  in  the  intensity 
of  the  images,  which  results  from  the  small  disturbance  in  tilted  angle.  However,  the 
Lambertian  images  show  very  little  difference  between  the  images.  This  means  that  the 
features  extracted  from  the  Lambertian  images  can  be  used  to  check  more  essential 
defects  such  as  missing  and  misplaced  components,  while  those  extracted  from  the 
sensitive  specular  images  are  used  to  check  more  sensitive  defects  such  as  tilted  and  not- 
properly-seated  components. 

It  has  been  reported  that  dozens  of  defect  types  can  be  found  in  real  PC  board 
manufacturing  lines  [Nav85].  In  this  work,  however,  we  have  attempted  to  identify  only 
those  defects  that  occur  most  often  in  practice.  According  to  the  information  provided  by 
industries  in  Florida,  the  most  important,  namely  the  most  frequently  arising,  component 
defects  are  missing,  misplaced,  tilted,  wrong  polarity,  and  not-properly-seated  components. 
Based  on  these  defects,  we  decided  to  design  a component  inspection  system  capable  of 
detecting  such  vision-oriented  defects  as  missing,  tilted,  and  not-properly-seated 
components,  since  such  electrical  characteristic -oriented  defects  as  misplaced  and  wrong 
polarity  components  can  be  easily  detected  using  an  in-circuit  tester,  which  is  now  widely 
used  in  most  electronic  plants  as  an  inspection  tool. 
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5.2  System  Overview 

In  Fig.  5.2,  we  show  a schematic  diagram  of  the  electronic  component  inspection 
system.  This  approach  is  based  on  three  monochromatic  images  (IjJ2J3)  obtained  from 
three  different  light  source  locations.  These  three  raw  images  are  separated  into  three 
Lambertian  images  (IUJ,2JI3)  and  three  specular  images  ( JtiJ&J «?)  using  the  normal  vector 
equalization  method  discussed  in  Chapter  3.  These  two  sets  of  images  consist  mainly  of 
Lambertian  and  specular  components  of  reflection,  respectively.  The  segmentation  data 
denoted  as  windows  in  the  block  diagram  are  incorporated  into  this  highlight  separation 
stage  to  process  the  images  only  for  the  areas  identified  as  components.  The  reflectance 
parameters  representing  the  optical  characteristics  of  the  component  surface  are  also 
provided  to  the  highlight  separation  stage.  They  are  measured  using  the  method  discussed 
in  Chapter  4.  These  window  data  and  reflectance  parameters  are  all  generated  in  the 
model  generation  phase. 

In  the  dual  channel  processing  scheme,  those  two  forms  of  image  information  are 
separately  processed  to  retrieve  a higher  level  of  information  most  effectively.  The  set  of 
Lambertian  images  is  averaged  and  the  set  of  specular  images  is  applied  into  a 
maximization  operator  to  obtain  an  image  in  which  every  highlight  observed  in  these 
specular  images  is  preserved.  Since  specular  images  usually  have  a bimodal  distribution 
of  the  grey-level  histogram,  a binarization  process  is  applied  to  reduce  computation  time 
for  further  processing.  The  thresholds  necessary  to  binarize  the  images  are  provided  from 
the  model  generation  phase  using  a model  PC  board. 
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Fig.  5.2.  Schematic  diagram  of  the  component  inspection  system. 
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In  order  to  separate  each  component  from  PC  board  background  in  the  images 
under  analysis,  a segmentation  method  based  on  the  concept  of  area  of  interest  [Pau90] 
is  employed.  Eventually  the  segmentation  process  can  be  completely  automated,  using  the 
locations  of  solder  joints  and  mechanical  dimension  of  components.  The  locations  of 
solder  joints  can  be  obtained  from  the  joint  registration  results  which  are  provided  from 
the  solder  joint  inspection  system  as  by-products  [Par89,  Par90a],  Using  the  segmentation 
data,  image  features  such  as  areas,  eccentricities,  and  average  intensities  are  extracted 
from  the  images  of  the  component  being  inspected.  These  features  are  compared  against 
those  of  stored  models  in  the  matching  process  using  a proximity  measure  which  is 
defined  by  normalized  absolute  error  between  the  features  of  a model  PC  board  and  those 
of  the  PC  board  under  inspection. 

After  the  matching  process,  the  system  can  determine  whether  the  underlying 
component  is  normal,  defective,  or  probably  defective  using  the  proximity  measure.  Based 
on  such  decision  results,  the  system  begins  classification  processes  in  which  the 
components  identified  as  defective  or  probably  defective  in  the  matching  process  are 
further  categorized  as  one  of  the  following  inspection  issues.  They  include  (1)  missing 
components,  (2)  not-properly-seated  components,  (3)  tilted  components,  and  (4)  defective 
components  of  an  unknown  class. 

Inspection  results  are  provided  in  two  forms  according  to  their  applications.  For 
touch-up  and  repair  work,  the  inspection  system  provides  touch-up  data  which  include 
only  the  locations  of  defective  components.  For  fault  diagnosis  and  process  control,  the 
defect  types  as  well  as  their  locations  are  provided  to  a fault  diagnosis  system  which  is 
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envisioned  to  combine  all  of  this  information  and  provide  feedback  to  selected 
manufacturing  processes  such  as  wave-soldering  and  component  insertion. 

5.3  System  Design 

The  design  of  a machine  vision  system  generally  involves  several  major  problem 
areas.  They  are  (1)  a sensing  and  preprocessing  problem  known  as  low-level  processing; 

(2)  a feature  extraction  and  scene  analysis  problem  known  as  intermediate-level 
processing;  and  (3)  a model  generation  and  matching  problem  known  as  high-level 
processing.  In  this  system,  we  have  employed  several  unique  methodologies  to  attack  the 
problem  areas.  They  are  (1)  a newly-developed  highlight  separation  technique  based  on 
the  normal  vector  equalization  method  for  the  preprocessing  problem  area,  (2)  the  dual 
channel  processing  scheme  for  feature  extraction  and  scene  description  problem  area,  and 

(3)  a proximity  measure-based,  decision-making  algorithm  for  the  matching  problem  area. 
We  will  discuss  the  details  of  the  system  design  in  terms  of  these  problem  areas. 

5.3.1  Sensing  and  Preprocessing 


As  in  many  other  automated  visual  inspection  systems,  the  first  step  in  designing 
whole  system  is  to  find  the  most  adequate  sensing  scheme.  In  most  cases,  a clever  front- 
end  can  greatly  reduce  the  complexity  of  higher-level  processes  such  as  feature  extraction 
and  matching.  In  this  work,  we  have  determined  to  employ  the  highlight  separation 
method  discussed  in  Chapter  3.  Therefore,  we  use  the  same  sensing  scheme  as  in  Section 
3.2.1.  The  imaging  system  presented  in  Fig.  3.4  is  also  used  to  digitize  and  process  the 
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PC  board  images.  In  order  to  measure  the  exact  values  of  the  source  vectors,  the  method 
discussed  in  Section  4.2.5  is  used  in  this  system. 

Based  on  this  imaging  system,  we  acquire  a set  of  original  images  which  contain 
both  Lambertian  and  specular  reflection  components.  These  images  are  applied  to  the 
highlight  separation  process  and  separated  into  two  sets  of  images  which  contain  only 
Lambertian  component  or  only  specular  component,  respectively.  The  normal  vector 
equalization  method  which  has  been  originally  developed  to  separate  probable  highlights 
and  obtain  3-D  shape  information  for  3-D  specular  objects  is  utilized  here  only  to  obtain 
the  sets  of  Lambertian  and  specular  images  of  a PC  board.  Even  though  the  normal  vector 
equalization  method  can  be  most  preferably  applied  to  a homogeneous  surfaces  which  are 
represented  by  a single  constant  value  of  reflectance  parameters  for  whole  surfaces,  the 
simulation  results  given  in  Section  3.5.2  allow  us  to  use  this  method  for  heterogeneous 
surfaces  like  the  components  on  a PC  board  by  setting  a single  representative  value  of  the 
parameters  [Par90b]. 

In  order  to  minimize  the  processing  time,  the  highlight  separation  process  is  only 
applied  to  the  window  areas  in  which  the  components  are  located.  The  window  data  are 
provided  from  the  segmentation  process  performed  during  the  model  generation  phase. 
To  determine  the  reflectance  parameters  for  each  segmented  area,  the  algorithms  discussed 
in  Chapter  4 have  been  employed  at  the  model  generation  phase.  Details  will  be  discussed 
in  the  subsequent  sections. 

The  dual  channel  processing  scheme  is  applied  to  these  sets  of  highlight-free 
(Lambertian)  images  and  highlight-only  (specular)  images.  The  set  of  Lambertian  images 
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which  can  be  utilized  to  obtain  3-D  surface  orientations  and/or  surface  shapes  [Par90e] 


is  here  applied  to  an  average  operator  which  is  defined  as: 


(5.1) 


where  Ila(x,y)  denotes  the  averaged  Lambertian  image  and  Iu(x,y)  denotes  the  ith 
Lambertian  image.  It  provides  very  reliable  information  about  the  existence  of  the 
component  under  inspection  as  discussed  in  Section  5.1. 

Whereas  the  Lambertian  images  are  applied  to  the  average  operator,  the  specular 
images  ( \IJx,y ),  i=12,3)  are  applied  to  a maximization  operator  to  find  an  image  in 
which  every  highlight  observed  in  each  individual  image  is  completely  conserved.  This 
image  is  denoted  as  Im(x,y)  and  obtained  as  follows: 


where  lj[x,y)  is  called  a maximized  specular  image.  All  kinds  of  the  specular  images 
d,i(x<y),  lAx,y),  f3(x,y),  Ijix.y))  are  applied  to  a binarization  process  as  follows: 


where  Itbfx,y)  and  l„,(x,y)  are  called  the  ith  binary  specular  image  and  a maximized  binary 
specular  image,  respectively.  Since  the  specular  images  usually  have  bimodal  distributions 
of  the  grey-level  histogram,  their  binarized  versions  (I*Jx,y),  IJx.y),  i=  1,2,3)  preserve 
most  information  originally  contained  in  the  corresponding  grey-level  images.  In  this 
work,  Otsu’s  algorithm  [Ots78]  has  been  employed  to  binarize  the  specular  images.  The 


Km^y)  = Maximum  of  \Isl(x,y)JJx,y)Js3(x,y)} 


(5.2) 


= Binary  version  of  {/,(x,y)}  , *=1,2,3 


(5.3) 


I*JW)  = Binary  version  of  f/im(x,y)} 


(5.4) 
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Otsu’s  algorithm  automatically  selects  an  optimal  threshold  level  so  as  to  maximize  the 
separability  of  two  classes  in  grey-level  histogram  using  the  discriminant  criterion 
[Fuk72].  The  algorithm  is  applied  to  each  individual  subimage  of  a component  determined 
from  the  segmentation  data  (windows). 

In  Fig.  5.3,  we  illustrate  examples  of  the  preprocessed  images  - original, 
Lambertian,  specular,  averaged  Lambertian,  maximized  specular,  binary  specular,  and 
maximized  binary  specular  images.  These  images  are  all  provided  to  the  feature  extraction 
stage  to  extract  as  many  useful  features  for  the  inspection  as  possible.  Throughout  this 
chapter,  we  will  use  the  same  PC  board  as  shown  in  Fig.  5.3  to  illustrate  the  operation 
of  each  processing  step. 

5.3.2  Segmentation 

In  order  to  supply  the  window  data  to  the  model  generation  phase  and  the 
inspection  phase,  it  is  necessary  to  extract  the  components  in  the  images:  this  problem  is 
called  image  segmentation.  It  is  usually  approached  by  detecting  edges  and  lines  or  by 
segmenting  the  image  into  distinct  regions.  Region  segmentation  focuses  on  the 
homogeneity  of  the  image  features,  using  the  assumption  that  those  features  will  be 
homogeneous  or  change  gradually  within  the  region  corresponding  to  each  component. 
However,  as  shown  in  Fig.  5.3(a),  the  original  image  of  an  assembled  PC  board  is  quite 
complex,  including  PC  patterns  and  many  shiny  components.  Therefore  it  seems  to  be 
very  difficult  to  segment  the  image  using  only  pure,  intensity  image-based  features  such 
as  edges  and  shading  [Ahn91].  Although  it  may  become  possible  after  huge  computational 
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Fig.  5.3.  Examples  of  the  preprocessed  images,  (a)  Original  image,  (b)  Lambertian  image, 
(c)  Averaged  Lambertian  image,  (d)  Specular  image,  (e)  Maximized  specular  image,  (f) 
Binary  specular,  (g)  Maximized  binary  specular  images. 
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Fig.  5.3.  - Continued. 
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efforts,  it  is  more  economical  to  implement  a segmentation  method  based  on  the  locations 
of  solder  joints  and  mechanical  dimension  of  the  components.  The  operations  of  the 
method  are  illustrated  in  Fig.  5.4(a).  While  running  the  program  developed  for  the 
segmentation  work,  an  operator  registers  the  window  points  (denoted  as  □)  which  consist 
of  the  left  upper  point  and  the  right  lower  point.  These  two  points  constitute  the  window 
area  within  which  features  are  extracted.  The  numbers  in  the  window  areas  denote  the 
identification  number  of  each  component. 

The  segmentation  data  contain  component  ED  number,  component  type,  and 
locations  of  the  window  points.  Most  of  these  data  can  be  automatically  obtained  from 
CAD  (Computer  Aided  Design)  data  and  mechanical  specifications  of  each  component 
if  the  PC  board  is  designed  using  an  appropriate  CAD  program.  The  window  points  can 
be  obtained  from  the  mechanical  dimension  of  the  component  and  a camera  model  which 
relates  the  dimension  given  by  a pixel  unit  in  image  plane  and  that  given  by  a metric  unit 
in  real  world.  In  Fig.  5.4(b),  we  present  the  segmentation  data  derived  from  the  example 
PC  board.  The  locations  of  the  window  points  are  described  using  the  image  coordinate 
system  in  which  the  origin  is  located  at  the  upper  left  comer  of  the  image,  the  x-axis  is 
given  by  the  vertical  line,  and  the  y-axis  is  given  by  the  horizontal  line. 

5.3.3  Feature  Extraction 

The  components  on  a PC  board  which  are  inspected  or  classified  have  a number 
of  discriminatory  attributes  or  features.  One  of  the  most  important  step  in  designing  the 
inspection  system  is  to  determine  what  discriminatory  features  to  select  and  how  to 
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Fig.  5.4.  Examples  of  the  segmentation  results,  (a)  Window  is  defined  as  the  inner  area 
spanned  by  the  two  window  points  denoted  as  □.  (b)  Segmentation  data. 
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measure  or  extract  these  features.  However,  in  most  automatic  inspection  problems  which 
arise  in  practice,  to  determine  complete  discriminatory  features  is  exceedingly  difficult 
and  highly-problem-oriented  task.  As  a result,  the  features  in  many  inspection  system  are 
determined  by  heuristic  methods  such  as  experimentation  and  trial-and-error  methods. 

The  features  employed  in  this  component  inspection  system  have  been  also 
determined  by  a number  of  experimentation.  They  are  largely  grouped  into  four  main 
categories;  (1)  average  intensity,  (2)  area,  (3)  eccentricity,  and  (4)  the  number  of  vertically 
and  horizontally  aligned  pixels.  The  basic  quantities  representing  these  main  categories 
are  defined  as  follows  and  used  to  describe  the  features  employed  in  this  system: 
Average  intensity  of  the  averaged  Lambertian  image  /,„  over  the  window  W. 

, y LM.  (5.5) 

NJW) 

where  I,a  denotes  the  averaged  Lambertian  image  defined  in  (5.1),  W denotes  the  window 
of  the  component  under  inspection,  and  Np(W)  denotes  the  number  of  pixels  within  the 
window  W.  Using  the  quantity,  we  define  the  first  feature  as  follows: 


F,  = A(IJ  (5.6) 

This  feature  is  very  useful  to  globally  constrain  the  component  under  inspection,  whether 
it  shows  strong  highlight  reflection  or  not.  The  feature  can  provide  good  cues  for  testing 
missing  components,  because  these  defects  are  often  characterized  by  an  abrupt  change 
of  the  average  intensity.  Moreover,  this  feature  can  provide  stable  information  for 
checking  the  above  failures,  because  they  are  extracted  from  the  averaged  Lambertian 
image  Ila,  which  is  insensitive  to  component  directions  and  lighting  conditions. 
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Area  of  an  object  in  the  binary  specular  image  over  the  window  W. 

S(Isbi)  = the  number  of  white  pixels  in  IM{x,y),  x,yzW  (5.7) 

where  IM  denotes  either  the  binary  specular  images  defined  in  (5.3)  (when  i=l,2J)  or  the 
maximized  binary  specular  image  defined  in  (5.4)  (when  i=m).  Several  features  are 
included  in  this  category.  These  features  use  specular  images  which  are  very  sensitive  to 
tilted  and  missing  components.  Therefore  the  features  are  very  useful  to  check  those 
defects  by  characterizing  the  mounting  status  of  many  shiny  components  such  as 
transistors,  capacitors,  and  resistors.  The  features  are  defined  as  follows,  using  the  binary 
specular  images  defined  in  (5.3)  and  (5.4): 

F,  - S(IM)  (5.8) 

F,  = W„2)  (5.9) 

F,  = S(IM)  (5.10) 

Ft  = S(/^,)  (5.12) 

Eccentricity  of  an  object  in  the  binary  specular  image  l.u.  over  the  window  W. 

E(Im)  = (5.12) 

AW*) 

where  IM  means  the  same  image  as  in  (5.7)  and  NH(Ishi)  and  NffJ  denote  the  maximum 
numbers  of  white  pixels  along  horizontal  and  vertical  directions  within  the  window  W, 
respectively.  In  Fig.  5.5,  we  illustrate  how  to  derive  Ny(IM),  Ny(IM),  and  eccentricity 
E(Lbi)  from  the  binary  specular  images  extracted  from  an  example  of  cylindrical 


window,  W 


N„ 

' N* 

highlight 

Fig.  5.5.  Determination  of  eccentricity. 
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components  such  as  capacitors  and  resisters.  is  obtained  by  performing  the 

following  procedures:  (1)  Scan  the  image  horizontally  while  counting  the  number  of  white 
pixels.  (2)  Repeat  the  first  procedure  while  increasing  the  horizontal  scan  line  vertically 
until  this  step  covers  the  entire  window  area.  (3)  Choose  the  maximum  number  of  white 
pixels  as  N^l^).  N^Ishi)  is  obtained  in  a similar  manner,  changing  only  the  scan  direction. 
The  features  in  this  category  are  defined  as  follows: 


f6  = mM) 

(5.13) 

ii 

(5.14) 

F*  = F{Im) 

(5.15) 

F9  = E(ISJ 

(5.16) 

These  features  are  especially  useful  to  check  either  tilted  or  not-properly-seated 
components,  because  they  sensitively  respond  to  the  variation  of  mounting  direction  of 
a component.  The  other  features  in  this  category  based  on  N^I^)  and  Ny(IM)  are: 


^.0  = W.J 

(5.17) 

Fn  = *h«,J 

(5.18) 

Examples  of  the  features  extracted  from  the  PC  board  shown  in  Fig.  5.3(a)  are 
illustrated  in  Fig.  5.6,  along  with  the  features  extracted  from  the  model  PC  board  shown 
in  Fig.  5.4(a). 
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Fig.  5.6.  (a)  Features  extracted  from  the  PC  board  shown  in  Fig.  5.3(a).  (b)  Features  extracted  from  the  model  PC  board  shown 
in  Fig.  5.4(a). 
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5.3.4  Model  Generation 

The  goals  of  the  model  generation  phase  are  (1)  to  derive  the  set  of  features  from 
a model  PC  board  to  which  the  set  of  features  extracted  from  a PC  board  being  inspected 
is  compared,  and  (2)  to  measure  reflectance  parameters  and  binarization  thresholds  to  be 
supplied  to  the  inspection  phase.  In  Fig.  5.7,  we  present  the  schematic  diagram  of  the 
model  generation  phase.  First  of  all,  a model  PC  board  satisfying  manufacturing  standards 
is  placed  at  a jig  which  controls  the  position  of  the  PC  board  within  the  field-of-view  of 
the  camera.  After  that,  the  raw  images  are  acquired  from  the  three  different  directions  of 
the  light  sources  as  discussed  in  Section  5.3.1.  Using  one  of  these  images,  the 
segmentation  process  is  performed  and  the  segmentation  data  represented  as  windows  are 
derived.  These  windows  are  supplied  to  the  other  processes  such  as  highlight  separation, 
reflectance  parameter  measurements,  preprocessing,  and  feature  extraction,  not  only  in  the 
model  generation  phase  but  also  in  the  inspection  phase.  This  is  to  minimize  the 
processing  time  by  applying  those  processes  only  to  the  window  areas  while  excluding 
the  background. 

In  order  to  estimate  the  reflectance  parameters  which  represent  the  optical 
characteristics  of  the  component  surface  in  each  window  area,  we  employed  the  parameter 
measurement  method  discussed  in  Chapter  4.  Based  on  the  assumption  that  the  underlying 
components  are  well-curved,  the  method  measures  the  most  essential  reflectance 
parameters,  R * and  K* , which  denote  a reflectance  constant  of  the  Lambertian  reflection 
component  and  a peak  intensity  of  the  specular  reflection  component,  respectively.  For 
the  surface  roughness  factor  m,  we  arbitrarily  determined  its  value  as  0. 1 using  the 
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Fig.  5.7.  Schematic  diagram  of  the  model  generation  phase. 
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experimental  results  given  in  Chapter  4,  since  most  of  the  components  in  a PC  board 
show  a medium  to  high  degree  of  specularity.  In  case  erroneous  parameters  are  measured, 
especially  for  planar  surfaces  or  highly  Lambertian  surfaces,  a manual  calibration  process 
is  performed  to  correct  the  data.  The  results  of  the  reflectance  parameter  measurements 
are  also  supplied  to  the  highlight  separation  processes  in  both  the  model  generation  phase 
and  the  inspection  phase. 

The  highlight  separation  processes  generate  the  Lambertian  images  and  the 
specular  images  from  which  the  features  discussed  in  Section  5.3.3  are  extracted.  A 
binarization  process  based  on  the  Otsu’s  algorithm  [Ots78]  is  performed  using  a model 
PC  board  to  obtain  the  binarized  versions  of  the  specular  and  maximized  specular  images. 
The  thresholds  obtained  from  the  binarization  process  are  supplied  to  the  preprocessing 
stage  in  the  inspection  phase  to  apply  the  same  threshold  levels  to  the  corresponding 
window  of  the  images  under  inspection. 

The  model  data  consist  of  component  ID,  type,  the  features  extracted  a model  PC 
board,  the  measured  reflectance  parameters,  and  the  thresholds  for  binarization.  Examples 
of  these  data  are  presented  in  Fig.  5.6(b),  Fig.  5.8(a),  and  Fig.  5.8(b),  respectively.  These 
data  are  all  extracted  from  the  model  PC  board  shown  in  Fig.  5.4(a). 

5.3.5  Inspection 

Under  exactly  the  same  imaging  conditions  as  the  model  generation  phase,  the 
inspection  processes  are  performed.  The  first  operational  step  in  the  inspection  phase  is 
to  place  a PC  board  to  be  inspected  at  the  same  location  using  a robotic  system  or  a jig 
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ID 

TYPE 

R* 

K* 

1 

R1 

38.0 

111.2 

2 

R1 

26.0 

136.1 

3 

R2 

22.0 

131.1 

4 

R1 

57.0 

119.0 

5 

Cl 

173.0 

2.0 

6 

TR1 

207.0 

48.1 

7 

R1 

15.0 

111.4 

8 

Cl 

233.0 

2.0 

9 

Cl 

29.0 

93.2 

(a) 


ID 

TYPE 

THRESHOLDS 

(i-l,2r3) 

1 

R1 

40 

42 

29 

2 

R1 

45 

42 

138 

3 

R2 

42 

40 

34 

4 

R1 

49 

53 

52 

5 

Cl 

39 

41 

88 

6 

TR1 

44 

10 

10 

7 

R1 

56 

53 

69 

8 

Cl 

82 

151 

101 

9 

Cl 

41 

155 

131 

(b) 


Fig.  5.8.  (a)  Reflectance  parameters  measured  from  model  PC  board  shown  in  Fig.  5.4(a). 
(b)  Threshold  data  extracted  from  the  same  PC  board.  These  are  applied  to  the  ith 
specular  image  obtained  from  the  PC  board  shown  in  Fig.  5.3(a). 
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control  mechanism.  The  next  step  is  to  take  three  raw  images  under  three  different 
locations  of  light  sources  and  proceed  to  the  other  processes  such  as  highlight  separation, 
preprocessing,  and  feature  extraction.  While  conducting  these  processes,  supplementary 
data  such  as  windows,  reflectance  parameters,  and  the  thresholds  for  binarization  are  all 
provided  from  the  model  generation  phase.  A matching  process  is  performed  to  find  the 
similarity  between  the  features  extracted  from  the  model  PC  board  and  the  PC  board 
under  inspection.  The  degree  of  the  similarity  is  determined  using  the  proximity  measure, 
P defined  as: 


nP  I IT 

p _ ^ 1 (5.19) 

^'i\model+^'i\insp 

where  F^moJel  and  F;]insp  denote  the  ith  feature  extracted  from  the  images  of  the  model 
PC  board  and  the  PC  board  under  inspection,  respectively.  NF  denotes  the  total  number 
of  features.  It  is  determined  as  1 1 in  this  system.  The  background  that  we  have  come  up 
with  (5.19)  to  measure  the  proximity  is  as  follows:  The  numerator  in  (5.19)  undoubtedly 
represents  the  closeness  between  the  features  extracted  from  the  model  PC  board  and  the 
PC  board  under  inspection.  The  denominator  serves  as  a normalization  factor  so  that  the 
resultant  proximity  measure  P is  invariant  to  the  absolute  amount  of  each  individual 
feature.  When  the  denominator  incidentally  becomes  to  zero,  we  can  ignore  the 
corresponding  features  and  make  their  contribution  to  P as  zero,  since  all  the  features  are 
designed  to  have  positive  values  and  as  a result  the  features  are  all  zero  in  this  case. 

Matching  criterion  for  inspecting  basic  assembly  status  is  to  check  whether  the 
proximity  measure  is  less  than  a threshold  Tml  or  greater  than  a threshold  Tm2.  That  is: 
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The  underlying  component  is 


normal  , if  P±Tml 
defective , if  P±Tm2 
probably  defective,  otherwise 


(5.20) 


In  order  to  determine  the  threshold  levels,  Tml  and  Tm2,  we  have  selected  a total  of  30 
sample  components  representing  the  defect  categories  of  our  concern.  Using  these  sample 
components,  we  carried  out  the  above  mentioned  processes  and  derived  their  histogram 
in  terms  of  the  proximity  measure  P.  The  resulting  histogram  is  presented  in  Fig.  5.9.  In 
this  histogram,  TmI  is  determined  as  the  value  of  P at  which  the  histogram  of  defective 
components  reaches  to  zero.  Similarly,  Tm2  is  determined  as  the  value  of  P at  which  the 
histogram  of  good  components  reaches  to  zero.  The  thresholds,  which  satisfy  these 
conditions,  are  determined  as  Tml= 1.7  and  Tm2= 3.5  from  the  histogram.  With  these 
thresholds,  we  can  minimize  the  defect  missing  rate  and  the  false  alarm  rate. 

By  adjusting  the  threshold,  however,  we  can  control  the  false  alarm  rate  or  defect 
missing  rate  further.  For  smaller  value  of  TmI,  we  can  reduce  the  missing  rate  and  make 
the  inspection  more  conservative.  Conversely,  for  larger  value  of  Tm2,  we  can  reduce  the 
false  alarm  rate  and  make  the  manufacturing  processes  more  productive.  Determining  the 
optimum  values  of  the  thresholds  is  crucial  in  real  applications. 

The  output  of  this  matching  process  becomes  the  touch-up  data  without  any 
modification,  because  they  include  only  the  locations  of  the  defects  and  probable  defects. 
However,  to  generate  the  fault  diagnosis  data,  we  apply  a tree-classifier  to  the  components 
identified  as  defects  and  probable  defects.  This  is  to  identify  the  categories  of  the  defect 
and  provide  this  information  to  a fault  diagnosis  system  which  is  envisioned  to  find  the 
most  plausible  reasons  for  the  defective  products.  The  identifiable  categories  of 
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1 ml  1 m2 


□ GOOD 
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Fig.  5.9.  Histogram  of  the  proximity  measure,  P derived  from  30  samples  of  the 
component  defects. 
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component  defects  include  (1)  normal,  (2)  missing,  (3)  tilted,  (4)  not-properly-seated,  and 
(5)  defective  components  of  an  unknown  class. 

The  tree-classifier  is  based  on  the  following  measures  derived  from  the  features 
defined  in  Section  5.3.3: 

(1)  DAS  (Difference  in  the  Area  of  Specular  images): 


DAS  = V'  IF  -F...  . 

/ ^ I 1 1 model  i\msp\ 


1-2 


(5.21) 


(2)  DAI  (Difference  in  the  Average  Intensity  of  Lambertian  images): 


DAI  - I'W'W  (5.22) 

(3)  DEC  (Difference  in  the  ECcentricity): 

8 2 IF  -F  I 

DEC  = V 1 ^model  '‘l^1  (5.23) 

• F a-F 

i\model  i\insp 

(4)  DHV  (Difference  in  the  number  of  white  Horizontal  and  Vertical  pixels): 

DHV  - Maximum  of  (|f10h^-f10|4J  , |Fllh-,-FM|^|  ) (5.24) 

These  measures  have  been  determined  by  numerous  experimentation.  They  serve  as  useful 
tools  to  make  classifications  for  the  components  identified  as  defective  or  probably 
defective  in  the  previous  processing  steps.  For  instance,  DAI  provides  reliable  cues  for 
classifying  missing  components,  DAS  is  used  for  tilted  and  not-properly-seated 
components,  and  DEC  and  DHV  is  used  for  detecting  normal  components  and  not- 
properly-seated  components. 
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In  Table  5.1,  we  present  the  measures  derived  from  the  30  sample  components 
which  have  been  used  to  determined  the  threshold  levels  in  the  matching  process.  Using 
these  data,  we  have  designed  the  tree-classifier  shown  in  Fig.  5.10.  The  input  of  the  tree- 
classifier  is  the  basic  inspection  results  given  by  the  forms  of  defect  and  probable  defect. 
The  component  identified  as  probable  defect  is  checked  again  using  the  measures,  DEC 
and  DHV.  If  these  measures  are  less  than  thresholds  T-,  and  Ta,  the  component  is  finally 
identified  as  a normal  component.  This  double-check  procedure  for  the  probable  defects 
can  reduce  the  false  alarm  rate  further  and  increase  the  productivity.  To  determine  the 
thresholds  Tn  and  Ti2,  we  need  to  analyze  distributions  of  the  measures  DEC  and  DHV 
for  normal  components  and  for  defective  components.  In  Table  5.1,  we  know  that  DEC 
and  DHV  of  most  normal  components  are  less  than  1.  Therefore,  Tit  is  determined  as  1.0. 
and  Ti2  is  determined  as  1.5  with  some  margin. 

To  the  components  identified  as  defective  in  the  matching  process  and  those 
having  failed  the  previous  two  tests  using  DEC  and  DHV,  we  apply  the  test  for  detecting 
missing  components  using  the  measure  DAI.  If  DAI  is  greater  than  a threshold  Ti3,  the 
defect  is  classified  as  a missing  component.  The  threshold  Ti}  is  determined  as  8,  because 
the  values  of  DAI  for  all  missing  components  are  less  than  8 as  given  in  Table  5.1.  The 
next  two  steps  are  to  detect  not-properly-seated  components  and  tilted  components.  These 
two  defects  show  very  similar  characteristics  in  all  the  measures  except  DHV  and  DAS. 
The  not-properly-seated  components  are  characterized  by  excessive  values  of  DHV  which 
is  greater  than  a threshold  Tj4.  Ti4  is  determined  as  25,  because  the  tilted  components  are 
characterized  by  the  measure  DHV  whose  values  are  all  less  than  25. 
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Table  5.1  The  measures  extracted  from  the  sample  components. 


class 

DAI 

DAS 

DEC 

DHV 

0 

283 

0.63 

0 

7 

36 

0.26 

3 

4 

337 

2.48 

0 

2 

44 

0.19 

0 

1 

4 

0.10 

0 

2 

14 

0.43 

0 

1 

113 

0.85 

0 

normal 

0 

24 

0.34 

1 

0 

14 

0.74 

0 

1 

14 

0.77 

0 

0 

10 

0.43 

0 

1 

0 

0.00 

0 

0 

5 

0.96 

0 

0 

3 

0.04 

1 

1 

6 

0.55 

1 

0 

8 

0.07 

1 

2 

185 

5.02 

8 

7 

1928 

5.87 

120 

n-p-sf 

2 

1125 

3.76 

91 

4 

81 

1.84 

12 

0 

58 

3.24 

20 

5 

94 

2.20 

24 

tilted 

5 

44 

3.20 

1 

3 

74 

5.30 

21 

6 

56 

4.20 

9 

12 

94 

6.00 

19 

missing 

9 

42 

4.00 

6 

9 

335 

2.23 

5 

10 

64 

1.81 

10 

t not-properly-seated 
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Fig.  5.10.  Block  diagram  of  the  tree-classifier  used  to  categorize  the  component  defects. 
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The  final  test  is  to  detect  the  tilted  components  from  the  defective  components 
which  failed  the  previous  testing  stages.  This  test  is  performed  using  the  measure  DAS, 
since  the  tilted  components  are  characterized  by  slightly  smaller  values  of  the  measure. 
If  DAS  is  less  than  a threshold  TiS,  the  defect  is  classified  as  a tilted  component.  The 
component  failing  this  test  is  classified  as  a defect  of  unknown  class.  The  threshold  Ta 
is  determined  as  60,  because  DAS  of  tilted  components  is  distributed  around  60  as  given 
in  Table  5.1. 

5.4  Experimental  Results 

Using  the  component  inspection  system  developed  in  the  PC-based  image 
processing  system  illustrated  in  Fig.  3.4,  we  have  performed  experiments  for  several  PC 
boards  obtained  from  industries  in  Florida.  The  entire  processes,  from  a PC  board  to 
inspection  results,  are  summarized  as  follows,  using  the  examples  shown  earlier: 

(1)  Acquisition  of  raw  images  from  PC  boards:  Fig.  5.3(a)  shows  one  of  the  raw 
images  taken  from  a PC  board  under  inspection.  Fig.  5.4(a)  shows  a similar  image 
taken  from  a model  PC  board. 

(2)  Preprocessing:  Fig.  5.3(b),  (c),  (d),  (e),  and  (f)  show  the  preprocessed  images 
extracted  from  the  PC  board  shown  in  Fig.  5.3(a).  From  the  model  PC  board 
shown  in  Fig.  5.4(a),  we  obtained  similar  images. 

(3)  Window  generation:  The  window  data  are  presented  in  Fig.  5.4.  This  process  is 
carried  out  in  the  model  generation  phase,  using  one  of  the  images  taken  from  the 


model  PC  board. 
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(4)  Reflectance  parameter  measurements:  The  reflectance  parameters  measured  for 
each  component  are  illustrated  in  Fig.  5.8(a).  This  process  is  also  carried  out  in 
the  model  generation  phase. 

(5)  Threshold  determination  for  binarization:  The  threshold  data  derived  in  the  model 
generation  phase  are  illustrated  in  Fig.  5.8(b). 

(6)  Feature  extraction:  The  features  extracted  from  the  PC  board  under  inspection  are 
illustrated  in  Fig.  5.6(a)  and  those  extracted  from  the  model  PC  board  are 
illustrated  in  Fig.  5.6(b). 

(7)  Inspection:  Examples  of  the  inspection  results  are  presented  in  Fig.  5.11. 
Inspection  results  can  be  displayed  on  the  monitor  screen  and/or  reported  in  print 
form  as  shown  in  Fig.  5.11(b).  The  printed  report  includes  the  ID  number,  type, 
location,  and  classification  result  of  each  component. 

We  repeated  the  processes  mentioned  above  for  15  PC  board  images.  The  statistics 
of  experimental  results  are  summarized  in  Table  5.2.  A total  of  149  components  was 
subjected  to  experimentation  and  an  average  correct  classification  rate  of  77.2%  was 
obtained.  The  results  are  divided  into;  (1)  (CC)  correct  classification  rate;  (2)  (G/B) 
correct  classification  rate  based  on  good/bad  point  of  view;  (3)  (FA)  false  alarm  rate;  and 
(4)  (M)  missing  rate.  These  performance  indexes  are  defined  as  follows: 

# of  components  correctly  classified 


CC  = 


# of  total  components 


(5. 25. a) 
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PCB1 

MODEL  INSPECTION 


(a) 


ID 

TYPE 

LOCATION 

INSP.  RESULT 

1 

R1 

( 64 

10) 

(122 

36) 

4 

2 

R1 

( 38 

116) 

( 94 

142) 

4 

3 

R2 

( 48 

142) 

( 82 

158) 

6 

4 

R1 

(120 

130) 

(144 

184) 

4 

5 

Cl 

(162 

184) 

(182 

216) 

6 

6 

TR1 

(148 

94) 

(196 

140) 

0 

7 

R1 

(168 

74) 

(216 

94) 

0 

8 

Cl 

(124 

50) 

(178 

72) 

0 

9 

Cl 

(178 

50) 

(232 

74) 

0 

0 :NORM,  3 :MISS,  4:N-P-S,  5:TILT,  6:UNK 


(b) 


Fig.  5.11.  Inspection  results,  (a)  Images  of  the  model  PC  board  and  the  PC  board  under 
inspection,  (b)  Printed  report  of  the  inspection  data. 
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Table  2.  Inspection  results. 


Image 

Total  Comp. 

Good  Comp. 

Bad  Comp. 

CC 

G/B 

FA 

M 

PCB1 

9 

6 

3 

6 

7 

2 

0 

PCB2 

5 

3 

2 

5 

5 

0 

0 

PCB3 

16 

9 

7 

13 

15 

1 

0 

PCB4 

9 

5 

4 

7 

7 

0 

2 

PCB5 

14 

10 

4 

11 

12 

1 

1 

PCB6 

13 

10 

3 

11 

12 

0 

1 

PCB7 

12 

9 

3 

9 

10 

2 

0 

PCB8 

6 

5 

1 

5 

5 

1 

0 

PCB9 

12 

5 

7 

9 

12 

0 

0 

PCB10 

9 

6 

3 

7 

8 

1 

0 

PCB11 

5 

3 

2 

3 

4 

1 

0 

PCB12 

15 

9 

6 

12 

15 

0 

0 

PCB13 

10 

5 

5 

7 

8 

1 

1 

PCB14 

8 

6 

2 

6 

8 

0 

0 

PCB15 

6 

0 

6 

4 

5 

0 

1 

TOTAL 

149 

91 

58 

115 

133 

10 

6 

TOTAL(%) 

77.2 

89.3 

10.9 

10.3 

CC:  Correct  Classification 
G/B:  Good/Bad  Classification 
FA:  False  Alarm 
M:  Defect  Missing 
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# of  comp,  correctly  classified  from  good/bad  point  of  view 


G/B= 


(5.25.b) 


# of  total  components 


# of  comp,  classified  as  bad  comp,  among  good  comp. 


FA  = 


(5.25.c) 


# of  total  good  comp. 


# of  comp,  classified  as  good  comp,  among  bad  comp. 


M 


(5.25.d) 


# of  total  bad  comp. 


Of  these  indexes,  the  missing  rate  deserves  particular  attention,  because  it  is  of  utmost 
importance  in  assuring  high  quality  products.  The  missing  rate  of  10.3%  denoted  in  Table 
5.2  implies  that  89.7%  of  the  defective  components  can  be  flagged  by  the  system. 


An  automated  inspection  system  for  electronic  components  assembled  on  a PC 
board  is  designed  and  implemented  in  a PC-based  image  processing  system.  In  designing 
the  system,  we  have  employed  several  unique  methodologies.  They  are  (1)  a highlight 
separation  technique  based  on  the  normal  vector  equalization  method,  (2)  an  intermediate- 
level  processing  scheme  known  as  dual  channel  processing,  and  (3)  a proximity  measure- 
based  matching  technique.  The  system  has  been  proven  to  work  properly  for  typical  PC 
boards  used  in  most  electronics  industries.  The  system  can  play  an  important  role  in 
automating  the  PC  board  assembly  processes,  not  only  as  an  essential  component  of  the 
PCB/AID  System,  but  also  as  a stand-alone  inspection  station. 


5.5  Summary 
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In  summary,  the  advantages  of  the  developed  system  are; 

(1)  It  can  detect  the  most  frequently  arising  defects  such  as  missing,  tilted,  and  not- 
properly-seated  components  on  average  PC  boards  which  satisfy  the  assumptions 
discussed  in  Section  5.3.2. 

(2)  It  can  take  and  process  images  of  up  to  approximately  5"X5"  PC  boards  in  a 
single  frame  of  512X512  resolution.  Therefore,  it  minimizes  the  number  of 
required  mechanical  movements  which  mainly  delay  the  system  processing  time. 

(3)  No  special  light  sources  are  required. 

The  disadvantages  are; 

(1)  The  size  of  the  required  memory  is  too  big,  since  it  must  store  the  sets  of 
Lambertian,  specular,  edge,  and  binarized  specular  images. 

(2)  Processing  several  frames  of  images  may  delay  the  overall  speed  of  the  system. 

(3)  A fine  mechanical  jig  is  required  to  control  the  locations  of  PC  boards. 

These  disadvantages  can  be  overcome  by  improving  either  software  or  hardware. 
For  instance,  a calibration  program  which  can  cancel  the  mechanical  drift  of  a jig  may 
be  used  to  compensate  for  the  mechanical  error.  Hardware  improvement  may  include  the 
use  of  more  semiconductor  memories  to  store  those  images  and  the  use  of  a multi- 
processing system  to  speed  up  the  system  performance. 

Further  suggestions  to  improve  the  system  performance  are  summarized  as 
follows: 

(1)  Development  of  discriminatory  features.  In  the  design  of  most  inspection  systems, 
determining  the  features  is  one  of  the  most  important  and  difficult  steps.  If  a 
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complete  set  of  discriminatory  features  for  each  defect  class  can  be  determined 
from  the  preprocessed  images,  the  classification  and  inspection  problem  will  be 
confronted  with  fewer  difficulties.  However,  in  most  inspection  problems,  the 
determination  of  such  features  is  an  extremely  difficult  task  and  it  can  only  be 
achieved  through  rigorous  experimentation. 

(2)  Determination  of  optimum  thresholds.  In  this  work,  the  thresholds  in  the  matching 
step  and  the  tree-classifier  have  been  determined  from  30  sample  components. 
Obviously,  the  statistical  data  extracted  from  more  sample  components  which 
represent  the  identifiable  categories  of  the  component  defects  can  assure  more 
accurate  thresholds.  To  this  end,  further  experimentation  with  various  defective 
components  is  essential. 

(3)  Automating  the  segmentation  processes.  The  segmentation  method  presented  in 
this  work  is  based  on  the  locations  of  the  solder  joints  and  the  mechanical 
dimensions  of  the  component.  However,  to  completely  automate  the  segmentation 
processes,  it  is  necessary  to  obtain  CAD  data  which  contain  the  component  ID, 
type,  and  location.  These  data  will  be  available  in  the  CIM  (Computer  Integrated 
Manufacturing)  environment  in  which  this  system  is  envisioned  to  operate. 


CHAPTER  6 
CONCLUSIONS 

6.1  Discussion 

This  dissertation  addresses  the  problems  concerning  highlight  separation, 
description,  and  inspection  for  3-D  specular  objects.  For  the  highlight  separation  problem, 
we  developed  the  normal  vector  equalization  method  which  is  based  on  the  empirical  light 
reflection  model  for  Lambertian  surface  and  the  simplified  Torrance-Sparrow  model  for 
specular  surface.  This  method  provides  a set  of  Lambertian  images  and  a set  of  specular 
images  from  common  monochromatic  images  which  are  acquired  from  three  different 
locations  of  light  sources.  These  two  sets  of  images  can  be  separately  applied  to  higher- 
level  processes  through  the  dual  channel  processing  scheme  to  extract  as  much  useful 
information  for  inspection  and  recognition  as  possible.  The  advantages  of  this  method 
over  the  previous  studies  are  summarized  as  follows: 

(1)  Its  application  areas  are  not  restricted  to  dielectric  materials.  It  can  be  applied  to 
general  surfaces  ranging  from  near  Lambertian  to  highly  specular. 

(2)  Common  monochromatic  images  are  utilized  in  this  method.  Therefore,  the  color 
of  the  light  source  does  not  need  to  be  different  from  that  of  the  surface  under 
application. 
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(3)  The  set  of  three  Lambertian  images  which  is  one  of  the  outputs  of  the  normal 
vector  equalization  method  can  be  used  to  derive  3-D  orientations  of  the  surface. 

(4)  No  mechanical  change  of  optical  filters  is  necessary.  Therefore,  all  of  the 
operational  processes  can  be  controlled  electrically. 

For  the  3-D  specular  object  description  problem,  we  used  the  results  of  highlight 
separation  to  derive  surface  orientations  and/or  3-D  depth  map  of  the  specular  object.  The 
surface  orientations  are  calculated  from  the  set  of  Lambertian  images  by  solving  the  light 
reflection  model  for  Lambertian  surface.  The  3-D  depth  map  is  derived  from  the  surface 
orientations  by  applying  the  depth  recovery  algorithm  which  is  based  on  line-integration 
technique.  The  advantages  of  the  surface  description  method  presented  in  this  dissertation 
are  summarized  as  follows: 

(1)  It  is  based  on  a simple  apparatus  for  sensing  images.  Therefore,  it  seems  to  be 
easy  to  implement  the  apparatus  in  real  applications  such  as  robotics  and  industrial 
inspection. 

(2)  It  provides  a fast  and  straightforward  algorithm  for  generating  surface  orientations 
and/or  3-D  depth  map.  Experimental  results  with  real  samples  prove  that  the 
developed  method  can  yield  accurate  reproduction  of  surface  orientations  and  3-D 
depth  map  in  which  no  unreasonable  surface  distortions  such  as  spikes  are 
observed. 

In  relation  to  the  above  two  problems,  we  also  developed  a method  for  measuring 
reflectance  parameters  which  characterize  the  optical  properties  of  the  given  surface.  The 
reflectance  parameters  must  be  known  a priori  in  the  highlight  separation  stage  to  apply 
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the  normal  vector  equalization  method.  This  method  is  based  on  the  strict  dependence  of 
highlight  upon  imaging  geometry.  It  first  measures  R*  which  represents  a reflectance 
constant  of  Lambertian  reflection  for  the  general  surface  in  which  those  two  reflection 
modes  are  present.  Secondly,  it  measures  fC  which  represents  peak  intensity  of  specular 
reflection  component.  Thirdly,  the  parameter  m which  represents  surface  roughness  is 
measured  using  a simple  fitting  method.  The  parameter  y which  represents  the  relative 
weighting  factor  between  Lambertian  and  specular  reflection  modes  can  be  measured 
using  a sample  of  perfectly  Lambertian  surface  or  perfectly  specular  surface.  Rough 
estimation  of  y can  also  be  obtained  using  the  value  of  /?’  and  K*.  This  method  can 
provide  accurate  measurements  of  the  reflectance  parameters.  Their  accuracy  and 
usefulness  in  the  highlight  separation  and  computer  graphics  are  proved  by  the 
experiments  carried  out  with  real  specular  objects. 

For  the  specular  object  inspection  problem,  we  have  applied  the  framework 
discussed  above  to  the  inspection  of  electronic  components  assembled  on  a PC  board.  The 
inspection  system  has  been  implemented  in  the  PC-based  image  processing  system.  The 
system  has  been  applied  to  real  PC  boards  manufactured  in  most  electronic  industries.  The 
advantages  of  the  system  are  summarized  as  follows: 

(1)  It  is  based  on  intensity  images  which  can  be  acquired  from  real  PC  boards 
quickly,  cheaply,  and  easily. 

(2)  It  minimizes  the  number  of  mechanical  movements  necessary  to  inspect  a whole 
PC  board  with  the  camera  having  limited  field-of-view. 

No  special  light  sources  are  required. 


(3) 
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The  methodologies  presented  in  this  dissertation  are  general  enough  to  be  applied 
to  many  other  fields.  The  normal  vector  equalization  method  can  be  applied  to  various 
machine  vision  problems  handling  3-D  specular  objects.  Typical  examples  include 
inspecting  surface  smoothness  of  industrial  specular  objects  and  recognizing  locations  and 
identities  of  these  objects. 

The  dual  channel  processing  scheme  can  also  be  applied  to  the  specular  object 
recognition  problem.  The  surface  orientations  or  3-D  depth  map  extracted  from  the 
Lambertian  images  can  be  used  to  globally  match  the  object  against  the  model  using,  for 
instance,  the  EGI  (Extended  Gaussian  Image)  matching  strategy  [Ike81b,  Bro84].  In  order 
to  reduce  search  space  in  the  matching  stage,  hypotheses  may  be  generated  using  the  2-D 
features  extracted  from  the  specular  images.  The  hypotheses  not  only  determine  the  most 
promising  candidates  among  the  models,  but  also  constrain  object  orientations  which  are 
inevitable  information  for  robotics  applications. 

The  method  for  measuring  reflectance  parameters  can  also  serve  as  an  important 
tool  for  such  fields  as  automatic  visual  inspection,  machine  vision,  and  computer  graphics. 
Many  of  the  algorithms  in  these  fields  are  based  on  physical  reflectance  models. 
Wherever  the  reflectance  models  are  concerned,  determining  the  reflectance  parameters 
becomes  a critical  problem  for  enhancing  overall  performance  of  the  system. 
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6.2  Contributions 

This  research  provides  satisfactory  solutions  for  the  fields  of  highlight  separation, 
description,  and  inspection  of  3-D  specular  objects.  The  key  contributions  of  this  research 
are  summarized  as  follows: 

(1)  Development  of  a versatile  highlight  separation  technique  which  we  call  the 
normal  vector  equalization  method.  This  method  utilizes  monochromatic  images, 
separates  probable  highlights  from  the  raw  images,  and  generates  Lambertian 
images  and  specular  images  [Par90b,  Par90e]. 

(2)  Developing  a conceptual  intermediate-level  processing  scheme  called  dual  channel 
processing.  In  the  dual  channel  processing  scheme,  the  two  different  aspects  of 
highlight  are  separately  used  to  maximize  the  system  performance.  In  one  channel, 
the  information  aspect  is  actively  exploited,  and  in  the  other  channel  the  distortion 
aspect  is  completely  avoided  [Par90c,  Pai91]. 

(3)  Development  of  a method  for  measuring  reflectance  parameters  of  a homogeneous 
surface.  This  method  utilizes  the  images  obtained  from  a sample  of  well-defined 
shape  such  as  a sphere  or  cylinder  and  measures  the  reflectance  parameters  and 
source  vectors  [Par90d], 

(4)  Development  of  an  automatic  inspection  system  for  electronic  components 
assembled  on  a PC  board.  This  system  has  been  designed  as  a practical  application 
of  the  framework  discussed  above  [Par90c,  Pai91].  The  system  works  as  a key 
component  of  the  PCB/AID  System. 


178 


(5)  Development  of  an  automatic  inspection  system  for  solder  joints.  This  system  has 
been  designed  as  an  integral  component  of  the  PCB/AID  System  [Par89,  Par90a]. 
Details  will  be  discussed  in  the  Appendix. 


APPENDIX 

THE  PCB/AID  SYSTEM 

(AUTOMATED  INSPECTION  AND  DIAGNOSIS  SYSTEM  FOR 
PRINTED  CIRCUIT  BOARD  MANUFACTURING) 

A.l  Introduction 

Automated  visual  inspection  technologies  came  to  the  PC  board  industries  in  the 
early  1980s  to  meet  an  urgent  need.  Driven  by  growing  market  demand  and  competition, 
automated  inspection  technology  quickly  became  an  industrial  standard.  Now,  the  PC 
board  industries  are  moving  toward  data-integrated  plant  in  which  the  design,  inspection, 
and  test  data  are  all  generated  from  a central  database  and  drive  the  various 
manufacturing  processes.  This  is  the  CIM  (Computer  Integrated  Manufacturing) 
environment,  and  automated  visual  inspection  is  becoming  an  integral  part  of  it. 

The  PCB/AID  System  has  been  conceived  to  accommodate  the  urgent  need  for 
higher  productivity  and  higher  quality  of  products.  The  schematic  diagram  of  the 
PCB/AID  System  is  illustrated  in  Fig.  A.l.  In  this  system,  PC  boards  are  applied  to  two 
inspection  stations;  one  is  for  solder  joints  and  the  other  is  for  electronic  components 
assembled  on  the  PC  boards.  The  results  of  the  two  inspection  stations  are  separated  into 
two  paths  according  to  their  applications.  For  touch-up  work,  the  inspection  system 
provides  touch-up  data  which  include  the  locations  of  solder  joint  defects  in  terms  of 
special  grid  coordinates.  For  the  applications  of  fault  diagnosis  and  process  control,  defect 
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manufacturing  processes 


Fig.  A.l.  Schematic  diagram  of  the  PCB/AID  System. 
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types  as  well  as  their  locations  are  provided  to  the  fault  diagnosis  stage  which  is 
envisioned  to  combine  all  of  the  inspection  information  and  make  feedback  to  selected 
manufacturing  processes  such  as  wave-soldering  and  component  insertion. 

During  the  last  decade,  a number  of  knowledge-based  systems  have  been 
developed  to  solve  diagnostic  problems.  Examples  include  the  MYCIN  system  [Sho75], 
the  MEDIKS  system  [Cha84,  Tou85b],  and  APRIKS  system  [Tou83].  Two  types  of 
elements  are  generally  used  for  diagnosis:  disease  and  manifestation.  In  the  PCB/AID 
System,  the  disease  means  faulty  processing  parameters  which  induce  faulty  products  and 
the  manifestation  denotes  the  inspection  results  obtained  from  the  faulty  products.  The 
goal  of  the  diagnostic  systems  is  to  find  the  most  plausible  diseases  from  the  observed 
manifestations. 

In  this  appendix,  we  will  restrict  our  attention  to  the  solder  joint  inspection  system. 
We  assume  that  an  appropriate  diagnosis  system  will  be  incorporated  with  the  two 
inspection  systems  to  complete  the  PCB/AID  System. 

A.  2 Automated  Solder  Joint  Inspection 
A.2.1  Statement  of  the  Problem 

In  electronics  industries,  soldering  is  the  primary  means  of  interconnecting 
components  mounted  on  PC  boards.  Visual  inspection  of  these  solder  joints  is  a crucial 
step  in  the  manufacturing  processes.  Automated  solder  joint  inspection  has  been 
considered  to  be  one  of  the  most  difficult  tasks  for  the  application  of  automated  visual 
inspection  techniques.  The  great  variability  of  solder  joint  defects  and  their  appearances 
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in  acquired  images  make  correct  classification  of  those  defects  even  more  difficult  [Par89, 
Par90a,  Bes85,  Bar88,  Cap88,  Sea85,  Juh86,  Rid90,  Nak85,  San88,  Ray88]. 

It  has  been  reported  that  dozens  of  solder  joint  defect  types  can  be  found  in  real 
PC  board  manufacturing  lines  [Nav85].  In  this  work,  however,  we  have  attempted  to 
identify  only  those  defects  that  occur  most  often  in  practice.  According  to  the  information 
provided  by  industries  in  Florida,  the  most  important,  namely  the  most  frequently  arising, 
solder  joint  defects  are  insufficient,  excessive,  and  bridge  solder  joint.  Besides  these 
defects,  our  system  has  been  designed  to  identify  other  fatal  defects  such  as  no-solder  and 
hole  joints  and  two  types  of  acceptable  joints.  Profiles  of  the  identifiable  solder  joints  are 
illustrated  in  Fig.  A.2. 

A.2.2  Review  of  Previous  Work 


Most  of  the  solder  joint  inspection  systems  announced  up  to  date  are  based  on  one 
of  the  following  technologies:  (1)  Visual  inspection,  (2)  Thermal  testing,  (3)  X-ray 
inspection,  and  (4)  3-D  laser  measurement.  Of  the  technologies,  most  of  the  academic 
work  has  been  accomplished  using  the  first  technology  because  of  their  deficiency  of 
sophisticated  illumination  sources  and  sensors  such  as  X-ray  or  lasers.  Besl  et  al.  [Bes85] 
have  implemented  a solder  joint  inspection  system  using  grey-scale  image  processing 
techniques.  Solder  joint  defects  are  identified  using  features  based  on  the  characteristics 
of  intensity  surfaces  and  a minimum-distance  classification  algorithm.  Bartlett  et  al. 
[Bar88]  announced  a solder  joint  inspection  system  mainly  modified  from  their  previous 
work  [Bes85].  Most  of  the  modification  has  been  placed  on  the  classification  section. 
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(c) 


<d) 


(e) 


(f) 


(g) 


Fig.  A. 2.  Profiles  of  the  identifiable  solder  joints,  (a)  Good,  (b)  Excessive,  (c)  Insufficient, 
(d)  Flat-normal,  (e)  No-solder,  (f)  Hole,  (g)  Bridge. 
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They  employed  two  classification  algorithms:  (1)  statistical  pattern  recognition  approach 
and  (2)  expert  system  approach.  Capson  and  Eng  [Cap88]  developed  a solder  joint 
inspection  system  using  color  image  analysis  techniques.  They  used  a tiered  lighting 
arrangement  to  generate  color  contours  on  solder  joint  images  and  exploited  features 
extracted  from  these  contours  for  the  detection  and  classification  of  defects. 

Many  of  the  commercial  systems  are  developed  using  sophisticated  illumination 
sources,  sensors,  and  mechanisms.  The  second  technology  is  employed  in  the  Vanzetti 
systems  [Sea85]  which  utilizes  infrared  signatures  of  solder  joints  for  inspection.  These 
signatures  are  produced  when  laser  heats  an  individual  solder  joint  and  a nearby  infrared 
sensor  records  its  thermal  radiation  curve.  The  third  technology  is  employed  by  IRT 
Corporation  [Juh86].  The  IRT’s  system  utilizes  radiographic  imaging  techniques  for 
obtaining  X-ray  images  of  solder  joints.  This  system  uses  shape  and  volume  properties 
of  the  X-ray  images  as  a means  of  classification.  The  fourth  technology  is  used  in  Robotic 
Vision  Systems’  laser-based  solder  joint  measurement  system  [Rid90].  The  system  utilizes 
structured-lighting  method  to  obtain  thousands  of  3-D  data  points  of  solder  joint  surfaces. 

The  other  attempts  include  (1)  Nakagawa’s  structured  light  approach  [Nak85];  (2) 
Sanderson’s  structured  highlight  approach  using  3-D  array  of  illumination  sources 
[San88];  and  (3)  Ray’s  solder  bump  inspection  system  using  a dark-field  and  bright-field 
illumination  scheme  [Ray88].  Performance  of  these  academic  systems  are  not  completely 
satisfactory  in  real  applications  and  most  of  the  commercial  systems  are  quite  expensive 
and  complex. 
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A.2.3  System  Overview 

In  Fig.  A. 3,  we  illustrate  the  schematic  diagram  of  the  developed  solder  joint 
inspection  system.  It  is  based  on  four  images  (I„  I2,  I3,  and  I4)  of  solder  joints:  Three 
images  are  obtained  using  multiple  point  sources  and  one  image  using  distributed  light 
sources.  From  these  images,  15  features  are  extracted  to  categorize  seven  classes  of  solder 
joints  - (G)  good  , (E)  excessive,  (I)  insufficient,  (N)  no-solder,  (H)  hole,  (F)  flat-normal, 
and  (B)  bridge  joints.  Data  about  locations  of  solder  joints  and  size  of  solder  pads,  which 
can  be  obtained  from  CAD  database,  must  be  provided  into  the  feature  extraction  stage. 
However,  for  cases  in  which  no  CAD  data  are  available,  an  automated  joint  registration 
system  has  been  developed  using  morphological  image  processing  techniques  [Par89]. 

The  system  first  classifies  six  categories  of  solder  joints,  excluding  only  bridge 
joints,  using  1 1 features.  The  next  classification  algorithm  runs  to  find  bridge  joints  only 
among  the  joints  classified  as  good  or  excessive  in  the  previous  classification  stage,  since 
bridge  joints  are  not  found  at  all  among  the  other  categories  of  solder  joints.  In  order  to 
find  bridge  joints,  six  features  are  used  in  the  second  classification  stage.  The  advantages 
of  employing  two  separated  classification  stages  are  (1)  it  reduces  processing  time, 
because  it  applies  the  bridge  joint  detection  algorithm  only  to  the  joints  which  could  be 
a part  of  bridge  joints;  and  (2)  it  becomes  much  easier  to  determine  classification 
parameters,  which  will  be  discussed  in  the  following  sections,  because  we  can  isolate  the 
bridge  classification  problem  from  the  whole  inspection  domain.  A classification 
algorithm  based  on  fuzzy  membership  functions  and  subjective  weightings  has  been 
developed  and  implemented  in  this  system. 
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Fig.  A.3.  Schematic  diagram  of  the  solder  joint  inspection  system. 
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Training  data  must  be  provided  at  the  classification  stages.  These  data  consist  of 
the  statistical  mean  and  standard  deviation  of  each  feature  of  each  solder  joint  class. 
Training  must  be  performed  in  the  exactly  same  condition  as  the  inspection  processes  are 
carried  out.  In  the  training  phase,  from  one  or  more  sample  PC  boards  which  contain  as 
many  representative  known  solder  joint  defects  as  possible,  four  images  are  taken  and 
processed  to  extract  the  features.  Then,  these  features  are  sorted  into  each  known  class 
and  their  statistical  mean  and  standard  deviation  are  calculated. 

Inspection  results  are  provided  with  two  forms  according  to  their  applications.  For 
touch-up  and  repair  work,  the  inspection  system  provides  a touch-up  tape  which  includes 
locations  of  detected  solder  joint  defects  in  terms  of  special  grid  coordinates.  For 
statistical  analysis  and  diagnostic  applications,  the  defect  types  as  well  as  their  locations 
are  provided  to  a statistical  analysis  stage.  An  on-line  diagnosis  system  is  envisioned  to 
combine  all  of  this  information  and  make  feedback  to  selected  manufacturing  processes 
such  as  wave-soldering  and  component  insertion. 

A.2.4  System  Design 

The  design  of  an  automatic  pattern  recognition  system  generally  involves  several 
major  problem  areas  [Tou76].  They  are  (1)  the  sensing  problem;  (2)  feature  extraction 
problem;  and  (3)  classification  problem.  We  will  discuss  the  details  of  the  system  design 
in  terms  of  the  three  problem  areas. 
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A.  2.4.1  Sensing 

As  with  many  other  automated  visual  inspection  systems,  it  is  the  first  step  in 
designing  a whole  system  to  find  the  most  adequate  sensing  scheme.  In  this  system,  we 
have  determined  to  employ  multiple  point  sources  from  four  directions  and  distributed 
sources  by  a number  of  experiments  and  measurements  for  various  solder  joint  defects 
in  various  conditions.  The  imaging  device  shown  in  Fig.  3.4  has  been  used  to  hold  the 
multiple  point  sources  and  a camera.  The  distributed  sources  are  simulated  using 
fluorescent  lights  installed  on  the  ceiling  in  our  laboratory.  Using  the  multiple  point 
sources,  three  images  are  obtained  under  three  different  elevation  angles  (0e)  of  20°,  45°, 
and  80°,  where  the  elevation  angle  is  defined  as  the  angle  between  a light  source  and  base 
plane.  We  denote  the  images  obtained  using  point  sources  as  I,  for  20°  elevation  angle, 
I2  for  45°,  and  I3  for  80°,  respectively.  The  image  obtained  from  distributed  sources  is 
denoted  by  I4  and  its  binarized  version  by  Is.  In  this  system,  Otsu’s  algorithm  [Ots78]  has 
been  employed  to  binarize  I4. 

The  background  that  the  multiple  point  sources  from  four  directions  are  employed 
in  this  system  is  explained  in  Fig.  A.4.  Essentially,  a highlight  in  an  image  occurs  in  such 
a way  that  surface  normal  vector  at  the  point  where  the  highlight  is  observed  bisects  the 
angle  between  the  light  incidence  vector  and  viewing  vector.  Therefore,  distances  between 
the  highlights  for  concave  surface  increase  as  illustrated  in  Fig.  A.4(a)  when  the  elevation 
angle  of  sources  increase  while  those  for  convex  surface  decrease  as  illustrated  in  Fig. 
A.4(b).  In  addition  to  this,  the  change  of  highlight  intensity  and  other  average  intensities 
over  several  different  regions  as  a function  of  the  elevation  angle  can  also  provide 
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(a) 


(b) 


S:  light  source  vector 
N:  surface  normal  vector 
V:  viewing  vector 
0e:  elevation  angle 

Fig.  A.4.  Relationship  between  highlights,  elevation  angle,  and  surface  convexity,  (a) 
Behavior  of  highlights  for  concave  surfaces,  (b)  Behavior  of  highlights  for  convex 
surfaces. 
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important  information  for  the  classification  of  various  solder  joint  defects.  The 
information  concealed  in  these  differences  of  highlight  locations  and  intensities  is 
effectively  utilized  in  the  feature  extraction  process  which  will  be  discussed  in  the  next 
section. 

In  detecting  bridge  solder  joints,  the  most  important  feature  is  connection 
information  between  adjacent  joints.  In  order  to  capture  this  information  most  effectively, 
we  need  an  image  in  which  whole  surface  of  solder  joint  can  be  discernible.  Obviously, 
we  can  not  obtain  such  an  image  using  the  point  sources  because  they  only  produce 
highlight  points  in  the  image  as  shown  in  Fig.  A.4.  The  only  choice  is  the  distributed 
sources  [Nay89]  because  they  emit  light  beams  into  solder  joint  surfaces  from  infinitely 
different  directions  covering  the  hemisphere  above  the  joint  surfaces  and  the  light  beams 
are  reflected  from  all  over  the  joint  surfaces  to  produce  such  an  image  that  shows  whole 
surfaces  of  the  joints. 

Examples  of  the  images  from  I,  to  I5  are  presented  in  Fig.  A.5.  As  discussed 
earlier,  I„  I2,  and  I3  show  bright  highlight  points  from  which  several  useful  features  are 
extracted  for  the  classification  of  solder  joint  defects  only  excluding  bridge  joints.  I5 
shows  most  of  solder  joint  surface  areas  and  represents  most  distinctly  the  connections 
between  bridge  joints. 

A.2.4.2  Feature  Extraction 

A total  of  15  features  is  used  to  recognize  seven  underlying  categories  of  solder 
joint  defects.  Of  the  15  features,  nine  features  are  prepared  to  classify  solely  good, 
excessive,  insufficient,  flat-normal,  no-solder,  and  hole  joints,  four  features  only  for  bridge 
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(a)  (b) 


(c)  (d) 


Fig.  A.5.  An  example  of  the  acquired  images,  (a)  I,,  (b)  I2.  (c)  I3.  (d)  Ij. 
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solder  joints,  and  two  features  for  both.  These  can  be  divided  into  three  main  categories: 
(1)  average  intensity  features,  (2)  highlight  features,  and  (3)  binary  image  connected 
region  features. 

Average  intensity  features.  Eight  features  are  included  in  this  category.  They  are 
defined  using  the  following  basic  quantity  which  represents  an  average  intensity  of  the 
kth  image  over  the  region  r: 


where  k denotes  the  image  under  processing,  r denotes  a summation  region  which  is 
illustrated  in  Fig.  A.6,  and  Np(r)  denotes  the  number  of  pixels  in  the  summation  region 
r.  In  Fig.  A.6.,  W denotes  the  region  of  a whole  solder  joint  subimage,  R and  C denote 
a rectangular  and  a circumference  region  within  a solder  joint  subimage,  and  L denotes 
a rectangular  region  covering  lead  area.  Using  (A.l),  we  have: 

(1)  F,  (fractional  change  of  average  intensity):  This  feature  is  extracted  from  I„  I2, 
and  I3.  It  quantitatively  measures  how  abruptly  the  average  intensity  measured 
from  I3  over  the  region  W changes  from  the  average  of  those  measured  from  I, 
and  Ij.  It  is  defined  as: 


E 


(A.l) 


Np(r) 


, £=1,2,3,  or  5 


Aj(W)+A2(W) 


(A.2) 


This  feature  is  especially  useful  for  the  classification  of  insufficient,  flat-normal, 
and  hole  joints  because  these  joints  are  almost  flat  for  most  areas  of  their  surfaces 
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Fig.  A.6.  Various  summation  regions,  (a)  W (whole  solder  joint),  (b)  R (rectangular),  (c) 
C (circumference),  (d)  L (lead). 
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and  therefore  they  are  characterized  by  a steep  increase  of  the  average  intensity 
when  the  illumination  angle  is  changed  from  20°  or  45°  to  80°. 

(2)  F2  (average  intensity  of  I,  over  L):  This  feature,  along  with  the  next  two  features 
(F3  and  F4),  is  extremely  useful  for  the  classification  of  no-solder,  hole,  and 
excessive  joints.  The  former  two  solder  joints  are  characterized  by  small  values 
of  all  three  features  while  the  latter  is  characterized  by  small  values  of  F2  and  F3 
and  relatively  high  values  of  F4.  It  is  defined  as: 

F2  = 4,(1)  (A.3) 

(3)  F3  (average  intensity  of  I2  over  L):  This  feature  is  obtained  by  applying  (A.3)  to 

h- 

F3  = A2(L)  (A.4) 

(4)  F4  (average  intensity  of  I3  over  L):  This  feature  is  obtained  by  applying  (A.3)  to 

I3. 

F4  = A3(L ) (A.5) 

(5)  Fs  (average  intensity  of  I3  over  R):  In  most  cases,  both  flat-normal  and  excessive 
joints  have  convex  surfaces  and  the  feature  extracted  from  highlight  distance  (F9) 
can  not  be  so  effective  in  differentiating  these  two  categories  of  joints  from  each 
other.  For  this  application,  this  feature  (F5)  has  been  provided.  It  provides  with  the 
larger  value  for  flat-normal  joints  than  excessive  joints  because  the  former  joints 
are  usually  flatter  than  the  latter  joints.  It  is  defined  as: 
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F,  = A3(R)  (A-6) 

(6)  F6  (average  intensity  of  I3  along  C)  : This  feature  is  used  to  categorize  good, 
excessive,  and  insufficient  solder  joints.  It  is  defined  as: 

Fs  = A,{C)  (A.7) 

The  radius  of  the  circumference  is  determined  as  some  fraction  of  the  radius  of 
solder  pad  which  is  provided  from  the  joint  registration  stage. 

(7)  F7  (average  intensity  of  I5  along  C)  : Essentially,  this  feature  is  the  same  as  F6 
except  that  this  is  extracted  from  I5. 

F1  = A, (C)  (A.8) 

(8)  Fg  (average  intensity  of  I3  over  W) : This  feature  is  used  to  find  the  bridge  solder 

joints  along  with  the  following  three  binary  image  connected  region  features.  It 
is  defined  as, 

Fg  = A3(W)  (A.9) 

Highlight  features.  Four  features  are  included  in  this  category.  Usually,  four 
highlights  can  be  found  in  I„  I2  and  I3  for  most  solder  joint  subimages,  as  illustrated  in 
Fig.  A.4  and  Fig.  A.5.  We  denote  the  ith  highlight  locations  observed  in  Ik  as  (Xi\Y,k), 
(kkN1,  where  Nk  denotes  the  number  of  highlights  in  the  kth  image  whose  intensity 
exceeds  a predefined  threshold  (T),  i.e.,  Ik(Xj\Y,k)>T  for  all  i.  Using  these  definitions  of 
highlight  locations,  we  define: 
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Average  distance  between  lead  and  highlights  in  the  kth  image. 


„ £ 1/(X0-X,V+(l'o-‘,.V  (A.  10) 

‘ ' h n* 

where  (Xo,Y0)  denotes  a lead  location  in  each  solder  joint  subimage,  which  is  provided 
from  the  joint  registration  stage. 

Average  highlight  intensity  in  the  kth  image. 


„ £ wW 

Hk  ~ 2L,  k 
m N* 


(A.11) 


where  Ik(X,k,Y|k)  denotes  intensity  of  the  ith  highlight  in  image  Ik.  Using  these  quantities, 
we  define  the  features  in  this  category  as  follows: 

(1)  F9  (difference  of  average  highlight  distance):  This  feature  is  extracted  from  I,  and 
12  and  represents  the  change  of  average  distance  between  highlights  and  lead 
locations.  It  is  defined  using  (A.  10)  as: 

F9  = DrD2  (A.  12) 

This  feature  is  especially  useful  for  the  determination  of  convexity  or  concavity 
of  solder  joint  surfaces.  Therefore,  this  feature  is  regarded  to  be  one  of  the  most 
reliable  features  for  the  classification  of  excessive,  good,  and  insufficient  joints. 
For  convex  surfaces  like  excessive  joints,  it  responds  with  positive  values  and  for 
concave  surfaces  such  as  insufficient  and  good  joints,  it  responds  with  negative 


values. 


197 


(2)  F10  (fractional  change  of  average  highlight  intensity):  This  feature  is  extracted 

from  I„  I2,  and  I3.  It  represents  how  abruptly  the  average  highlight  intensity 
measured  in  I3  changes  compared  to  the  average  of  those  measured  in  I,  and  fy 
This  feature  is  defined  using  (A.  11)  as: 


fio  = 


2 H3 

~h^h2 


(A.13) 


(3)  Fjj  (the  number  of  highlights  whose  intensity  is  less  than  a threshold):  Although 
four  highlights  can  be  observed  for  most  of  good,  excessive,  and  insufficient 
joints,  in  some  cases  in  which  no-solder,  hole,  and  flat-normal  joints  are  involved, 
the  number  of  highlights  can  be  less  than  four.  Therefore,  this  feature  can  be 
utilized  to  classify  these  joints  from  the  rest.  The  threshold  of  highlight  intensity 
is  dependent  on  imaging  parameters  such  as  illumination  source  intensity  and 
sensitivity  of  camera.  This  feature  is  extracted  from  I,  and  defined  as: 


Fn  = 4 -Nl  (A-14> 

(4)  F12:  This  feature  is  in  essence  the  same  as  Fn  except  that  F12  is  extracted  from  fy 

That  is, 

Fu  = 4 -N2  (A.  15) 

Binary  image  connected  region  features.  Three  features  are  included  in  this 
category.  These  features  are  exclusively  used  to  detect  bridge  joints  and  extracted  from 
I5  for  the  area  between  two  adjacent  joints.  As  illustrated  in  Fig.  A.7,  two  quantities  (NB 
and  Nw)  are  measured  within  a window  which  is  determined  using  lead  location  and  pad 
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Fig.  A.7.  Binary  image  connected  region  features  extracted  from  a bridge  joint  image. 
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size  data  provided  from  the  joint  registration  stage.  NB  (or  Nw)  denotes  the  number  of  8- 
connected  lines  which  consist  of  consecutive  black  (or  white)  pixels  whose  length  exceeds 
a predefined  threshold.  The  threshold  is  determined  to  be  some  fraction  of  the  distance 
between  two  adjacent  joints.  Using  these  quantities,  we  have: 

(1)  F,3  (The  number  of  consecutive  black  lines  between  joints): 

F„  = Nt  (A.16) 

(2)  F14  (The  number  of  consecutive  white  lines  between  joints): 

Fu  - Nw  (A.17) 

(3)  F,j  (Addition  of  F13  and  Fu)  : By  adding  F13  and  F14,  this  feature  is  obtained.  It 
is  also  used  to  find  bridge  joints. 

F„  - nb'nw  <A18> 

A.2.4.3  Classification 

We  employed  fuzzy  set  theory  [Zad65]  to  develop  a classification  algorithm  which 
can  categorize  underlying  solder  joint  defects  with  maximum  accuracy.  The  advantage 
provided  by  fuzzy  sets  is  that  the  degree  of  membership  in  a set  can  be  specified,  rather 
than  just  the  binary  "to  be  or  not  to  be"  a member.  This  can  be  especially  advantageous 
in  a pattern  recognition  problem,  where  frequently  objects  are  not  clearly  members  of  one 
class  or  another  as  the  solder  joint  recognition  problem.  Using  crisp  techniques,  an 
ambiguous  object  will  be  assigned  to  one  class  only  under  any  situation.  On  the  other 
hand,  fuzzy  techniques  will  specify  to  what  degree  the  object  belongs  to  each  class,  which 
is  useful  information  in  many  cases  [Kel85a,  Kel85b]. 
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In  this  section,  we  will  first  introduce  the  classification  algorithm  developed  using 
fuzzy  membership  functions  and  subjective  weighting  factors,  and  then  discuss  other 
technical  details  which  are  essential  to  understand  the  algorithm. 

Classification  algorithm.  Let  X be  a solder  joint  described  by  15  features,  then 
X={F„  ... , FIS}.  For  each  solder  joint  class  Wt,  let  U,j(Fj)e  [0,1]  be  a membership  function 
for  the  jth  feature  of  the  ith  class  of  solder  joint  defects.  Then,  for  each  feature  Fj,  U,j(Fj) 
measures  the  membership  of  X in  W,  from  the  standpoint  of  a single  feature  value  Fj. 
This  partial  evaluation  of  the  degree  of  membership  is  combined  with  the  subjective 
weighting  factors  gtj  which  expresses  the  extent  to  which  a viewpoint  of  feature  Fj  is 
important  in  evaluating  a solder  joint  X from  the  class  Wj.  The  following  measure  E, 
gives  an  evaluation  of  the  degree  to  which  a solder  joint  X belongs  to  class  W,: 

n 

Ei  = £ andj=  (A19) 

7-i 

where  n and  m denote  the  number  of  features  and  the  number  of  defect  classes, 
respectively.  The  combination  of  these  two  numbers,  (n,m)  becomes  (11,6)  for  the  first 
classification  stage  and  becomes  (5,3)  for  the  second  stage.  Final  decision  for  the 
classification  of  solder  joint  defects  is  made  in  such  a way  that  a solder  joint  belongs  to 
class  Wj  if  and  only  if  E,  is  greater  than  Ek  for  all  k*i. 

Subjective  weighting  factors.  The  subjective  weighting  factor  g,j  is  determined 
heuristically  using  the  following  constraints  and  provided  into  (A.  19): 
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E*  = i (A20a) 

gy  e [0,1]  (A.20.b) 

These  constraints  are  used  to  impose  uniform  weightings  between  each  different  class. 
The  procedures  for  determining  the  weighting  factors  are  (1)  to  determine  initially  each 
value  of  the  factors  by  intuition  using  (A.20);  (2)  to  check  classification  results  and  find 
whether  there  is  any  bias  for  being  classified  to  specific  classes;  (3)  to  adjust  the  bias  by 
changing  the  weighting  factors  for  the  features  and  the  classes  involved  in  the  bias;  and 
(4)  to  continue  the  step  (2)  and  (3)  until  arriving  at  successful  results.  Ultimately,  two  sets 
of  weighting  factors  for  each  classification  stage  have  been  determined  as  Table  A.l  by 
using  the  above  procedures. 


Table  A.l.  Weighting  factors  (gy).  (a)  gy  for  the  first  classification  stage,  (b)  g^  for  the 
second  classification  stage. 


(a) 


j 

1 

2 

3 

i 

*i 

f2 

f3 

1:6 

0.1 

0.0 

0.0 

2 :E 

0.1 

0.0 

0.0 

3:1 

0.3 

0.0 

0.1 

4 : F 

0.2 

0.0 

0.0 

5 :N 

0.0 

0.1 

0.4 

6 :H 

0.0 

0.1 

0.3 

4 5 6 7 
F4  f5  f6  f7 

0.0  0.0  0.2  0.3 
0.0  0.0  0.3  0.4 
0.0  0.0  0.3  0.0 
0.3  0.2  0.0  0.0 
0.1  0.2  0.2  0.0 
0.3  0.1  0.2  0.0 


8 

9 

10 

11 

f9 

Fj.o 

Fu 

Fl2 

0.4 

0.0 

0.0 

0.0 

0.2 

0.0 

0.0 

0.0 

0.1 

0.2 

0.0 

0.0 

0.0 

0.2 

0.1 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

(b) 


1:6 
2 :E 
3 : B 


1 2 3 4 5 6 

^6  f7  f13  f14  f8  f15 

0.2  0.2  0.1  0.1  0.2  0.2 

0.2  0.1  0.2  0.2  0.1  0.2 

0.2  0.2  0.1  0.1  0.2  0.2 
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Fuzzy  membership  functions.  In  order  to  evaluate  (A.  19),  we  also  need  to  define 
the  membership  functions,  U^Fj).  Two  standard  functions  called  S-function  and  rc-fimc- 
tion  are  used  to  define  the  membership  functions.  They  are  illustrated  in  Fig.  A.8  and 
mathematically  defined  as  [Pal86]: 


S(q;a,b,c ) = 0 

, qza 

, 2 <«  "a)2 

, a<qzb 

{c-ay 

- 1-2  <*-#, 

, b<qzc 

ic-af 

= 1 

, c<q 

(A.21) 


7i  (q;a,b,c)  = S(q;a,b,c)  , q<c 

(A.22) 

= \-S(q;c,2c-b,3c-2b)  , q±c 

where  b denotes  a cross-over  point  at  which  S(b;a,b,c)=7t(b;a,b,c)=0.5,  and  b=(a+c)/2.  In 
7t(q;a,b,c),  c denotes  the  central  point  at  which  tc=1,  and  2(c-b)  is  the  bandwidth,  i.e.,  the 
separation  between  the  cross-over  points  of  the  Tt-function.  (A.21)  and  (A.22)  define  the 
membership  functions  corresponding  to  fuzzy  sets  ’q  is  large’  and  ’q  is  c\  respectively. 
Using  these  functions,  we  defined  three  types  of  the  membership  functions  employed  in 
this  system: 


type- 1;  U^F-l)  = S(Fra^c.)  (A.23.a) 

type -2;  U^F- 2)  = 1 -5(F.;flv,b.,c.)  (A.23.b) 


type-3;  U^Fp)  = n (F^ap^c^) 


(A.23.C) 
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(a) 


20  40  SO  SO  100 


(b) 

Fig.  A.8.  S-function  and  rc-function.  (a)  S-function  (a=30,  b=50,  c=70).  (b)  7C-function 
(a=30,  b=40,  c=50) 
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Table  A.2.  Type  of  membership  functions  (ty).  (a)  t,j 
t|j  for  the  second  classification  stage. 


(a) 


j : 


1 : G 
2 :E 
3:1 
4 : F 
5 :N 
6 : H 


1 

Fx 

3 

3 

1 

1 

3 

3 


2 3 4 

f2  f3  f4 

3 3 3 

3 3 3 

3 3 3 

3 2 1 

2 2 3 

2 2 2 


5 6 

f5  f6 

3 3 

2 2 
3 3 

1 1 
1 1 
3 3 


(b) 


1:6 

2 :E 

3 :B 


1 2 
f6  f7 

1 1 
2 2 
3 3 


3 

F13 

3 

1 

3 


for  the  first  classification  stage,  (b) 


7 8 9 10  11 

F7  F,  f10  fu  f12 

3 2 3 2 2 

2 13  2 2 

3 2 12  2 

13  111 

11111 
1 1 3.  1 1 


4 5 6 

f14  f8  f15 

3 12 

2 2 3 

13  1 


The  type  of  a membership  function  is  essentially  determined  by  characteristics  and 
natures  of  features  in  each  pattern  class.  The  type-1  (type-2)  membership  function  serves 
as  a window  for  the  features  Fj  which  give  more  certainty  of  being  categorized  into  the 
class  W,  for  greater  (smaller)  value  of  the  features.  For  instance,  more  positive  value  of 
F9  extracted  from  a solder  joint  gives  more  certainty  of  being  categorized  into  an 
excessive  joint.  Therefore,  we  select  the  type-1  membership  function  to  evaluate  the 
degree  of  membership  for  an  excessive  joint  from  the  standpoint  of  F9.  On  the  other  hand, 
we  select  the  type-2  membership  function  for  good  or  insufficient  joints  because  these 
joints  are  characterized  by  negative  or  less  values  of  F9.  The  type-3  membership  function 
is  provided  for  the  features  which  give  more  certainty  for  being  categorized  into  a class 
when  their  values  are  closer  to  their  statistical  mean.  For  instance,  in  evaluating  a good 
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joint  using  F„  the  type-3  membership  function  is  selected  because  good  joints  are 
characterized  by  mid-range  values  of  this  feature.  In  Table  A.2,  we  present  the  types  of 
membership  functions  which  have  been  determined  using  the  calibration  procedures 
discussed  above.  The  elements  in  Table  A.2  are  denoted  by  ty  to  illustrate  the  type  of  the 
membership  function  U,j(Fj). 

After  the  type  of  a membership  function  is  determined,  the  parameters  a^,  by,  and 
c,j  in  (A. 23)  which  determine  the  shape  of  the  membership  functions  are  adjusted  as 
follows  using  statistical  data  provided  by  the  training  phase.  These  data  are  denoted  by 
my  and  stj  which  indicate  statistical  mean  and  standard  deviation,  respectively,  of  a feature 
Fj  extracted  from  a known  ith  class  of  solder  joints. 


(A.24. a) 

i 

O 

II 

-e? 

(A.24.b) 

dy  - 2 by  Cy 

(A.24.C) 

where  Z denotes  a fuzzifier  which  determined  the  fuzziness  of  a membership  function. 
When  Z=0,  the  membership  functions  become  crisp  characteristic  functions  whose 
boundary  is  c.  In  our  system,  Z is  determined  as  four  after  performing  a number  of 
experiments. 

The  rationale  that  the  parameters  in  membership  functions  are  adjusted  using  the 
statistical  data  as  in  (A.24)  originates  from  the  following  observations: 

(1)  A statistical  mean  of  features  implies  a dc-offset  of  the  samples  in  the  feature 
space.  Therefore,  it  can  be  utilized  to  determine  a reference  point  of  the 
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corresponding  membership  function.  In  this  system,  we  determined  the  reference 
point  as  Cy  and  employed  (A.24.a)  to  coincide  the  peak  point  of  the  membership 
function  with  the  statistical  mean. 

(2)  A standard  deviation  of  features  denotes  how  widely  the  samples  are  spread  in  the 
feature  space.  Therefore,  it  can  be  used  to  determine  the  fuzziness  of  a 
membership  function.  In  this  system,  the  cross-over  point  by  has  been  selected  to 
determine  the  fuzziness  in  the  membership  function  by  incorporating  the  standard 
deviation  with  the  fuzzifier  Z using  (A.24.b). 

In  conclusion,  the  membership  function  for  the  jth  feature  of  the  ith  class, 
U|j(Fj;t1j,mlj,s1j)  is  determined  by  three  factors:  One  is  the  type  (tfj)  of  the  function  presented 
in  Table  A.2  and  the  others  are  the  statistical  data  (m^s,.)  measured  from  the  training  sets 
of  the  solder  joints  whose  classes  are  known  as  a priori  information.  The  value  of  the 
membership  function  measured  using  a feature  Fj  is  applied  to  (A.  19)  in  conjunction  with 
the  subjective  weighting  factors  which  are  determined  as  Table  A.l.  The  measure  E,  in 
(A.  19)  gives  a combinational  degree  of  membership  of  a solder  joint  belonging  to  a defect 
class  W,  and  constitutes  a basis  for  the  classification  of  solder  joint  defects  and  generation 
of  diagnostic  data  and  touch-up  tapes. 

Rules  for  touch-up  tape  generation.  Classification  results  obtained  by  finding 
maximum  of  E,  (denoted  as  Max(Et))  in  (A.  19)  can  be  used  for  diagnostic  data  without 
modification.  However,  for  touch-up  tape  generation,  following  rules  are  applied  to  the 
classification  results  obtained  by  (A.  19)  to  reduce  defect  missing  rate  further: 
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(1)  IF  Max(E,)  denotes  defects,  THEN  flag  the  joints  as  defects. 

(2)  IF  Max(E,)  denotes  good  or  flat-normal  joints  and  difference  between  Max(E,)  and 
the  Second  Max(E,)  is  less  than  a predefined  threshold,  THEN  flag  the  joints  as 
probable  defects. 

(3)  ELSE  flag  the  joints  as  acceptable  joints. 

The  joints  classified  as  probable  defects  by  these  rules  are  checked  again  in  a 
touch-up  process  by  human  inspector.  By  following  this  double-check  inspection  strategy, 
we  can  ensure  that  most  of  solder  joint  defects  on  a PC  board  can  be  repaired,  although 
increased  false  alarm  rate  may  deteriorate  productivity  more  or  less. 

A.2.5  Experimental  Results 

Experiments  have  been  performed  using  several  real  PC  boards  obtained  from 
industries  in  Florida.  Since  no  CAD  data  about  joint  locations  and  size  of  pads  are  avail- 
able for  these  PC  boards,  manual  registrations  of  solder  joints  are  performed  using  a user- 
interface  program.  The  results  of  joint  registrations  include  ED  numbers,  locations,  and 
pad  size  of  each  solder  joint. 

Classification  results  can  be  displayed  on  a monitor  screen  and/or  reported  in 
printed  forms.  The  printed  reports  for  diagnostic  applications  include  ID  numbers  of 
solder  joints  and  the  PC  board  under  inspection,  joint  locations,  derived  feature  values, 
and  recognized  categories.  Touch-up  tapes  for  repair  work  are  generated  by  applying  the 
above  extra-rules  to  the  diagnostic  data. 
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Table  A.3.  Solder  joint  classification  results. 


PCB 

Image 

Total 

joints 

Good 

joints 

Bad 

joints 

CC 

G/B 

FA 

M 

A 

A1 

28 

13 

15 

27 

28 

0 

0 

A2 

33 

17 

16 

30 

30 

3 

0 

A3 

28 

21 

7 

28 

28 

0 

0 

A4 

28 

14 

14 

26 

27 

1 

0 

Subtotal 

117 

65 

52 

111 

113 

4 

0 

B 

B1 

80 

42 

38 

76 

76 

4 

0 

B2 

78 

40 

38 

74 

75 

2 

1 

B3 

61 

30 

31 

58 

58 

1 

2 

B4 

78 

26 

52 

70 

73 

1 

4 

Subtotal 

297 

138 

159 

278 

282 

8 

7 

C 

Cl 

80 

66 

14 

72 

72 

7 

1 

C2 

60 

36 

24 

58 

60 

0 

0 

C3 

68 

58 

10 

65 

65 

2 

1 

Subtotal 

208 

160 

48 

195 

197 

9 

2 

D 

D1 

110 

30 

80 

103 

105 

1 

4 

Subtotal 

110 

30 

80 

103 

105 

1 

4 

Total 

732 

393 

339 

687 

697 

22 

13 

Total (%) 

93.9 

95.2 

5.6 

3.8 

CC;  Correct  Classification 
G/B;  Good/Bad  Classification 
FA;  False  Alarm 
M;  Missing 
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In  Table  A.3,  we  present  the  recognition  results  obtained  by  applying  the  PC 
boards  to  this  system.  A total  of  732  joints  was  experimented  and  an  average  correct 
classification  rate  of  93.9%  was  obtained  using  the  classification  criterion  described  in 
(A.  19).  The  results  are  divided  into;  (1)  (CC)  correct  classification  rate;  (2)  (G/B) 
good/bad  classification  rate;  (3)  (FA)  false  alarm  rate;  and  (4)  (M)  missing  rate.  These 
performance  indices  were  defined  in  Chapter  5.  Of  these  indices,  the  missing  rate 
deserves  particular  attention,  because  it  is  of  utmost  importance  in  assuring  high  quality 
products.  The  missing  rate  of  3.8%  denoted  in  Table  A.3  implies  that  96.2%  of  defective 
joints  can  be  flagged  by  the  system  without  using  the  extra-rules  for  touch-up  tape 
generation. 

A.2.6  Summary 

An  automated  solder  joint  inspection  system  is  designed  and  implemented  using 
FORTRAN  77  in  VAX1 1-750/VMS  computer.  The  system  can  serve  as  an  integral  part 
of  the  PCB/AID  System.  The  inspection  system  utilizes  four  frames  of  solder  joint 
images,  extracts  15  features  from  the  images  to  categorize  the  most  important  seven 
classes  of  solder  joint  defects,  and  provides  two  forms  of  inspection  results  according  to 
their  applications.  The  system  has  been  proven  to  work  properly  for  average  PC  boards 
used  in  most  electronics  industries.  The  system  will  play  an  important  role  in  automating 
the  PC  board  manufacturing  processes,  not  only  as  an  essential  component  of  the 
PCB/AID  System,  but  also  as  a stand-alone  inspection  station. 
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In  summary,  the  advantages  of  our  system  are; 

(1)  It  is  based  on  intensity  images  which  can  be  acquired  from  real  PC  boards  quickly 
and  cheaply. 

(2)  It  can  classify  the  seven  most  important  categories  of  solder  joints,  including 
bridge  solder  joints,  with  a high  recognition  rate. 

(3)  It  can  process  images  of  up  to  250  solder  joints  in  a single  frame  of  512  by  512 
resolution.  Therefore,  it  minimizes  the  number  of  required  mechanical  movements 
which  mainly  delay  the  system  processing  time. 

(4)  No  special  light  sources  are  required. 

(5)  According  to  applications,  it  can  provide  two  different  types  of  inspection  results; 
one  is  touch-up  tape  and  the  other  is  diagnostic  data. 
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